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WHAT WORKS? 
COMMON PRACTICES IN HIGH FUNCTIONING AFTERSCHOOL PROGRAMS: 
THE NATIONAL PARTNERSHIP FOR QUALITY AFTERSCHOOL LEARNING 
FINAL REPORT 


Abstract 


In an effort to identify and incorporate quality practices into existing and future 
afterschool programs, the U.S. Department of Education commissioned the National 
Partnership for Quality Afterschool Learning (National Partnership) to study high 
functioning 21° Century Community Learning Center programs. The purpose of this 
study was to document practices to be used to develop resources and professional 
development that address issues relating to the establishment and sustainability of 
afterschool programs, to provide models and indicators of promising practices, and to 
highlight other descriptive information that local sites can access in planning new 
afterschool programs or improve existing ones. Fifty-three afterschool programs, 
representing eight regional divisions of the nation, including rural and urban programs 
and community-based and school district related programs, were identified using 
rigorous methods. Promising practices in content delivery, program organization, and 
program structure were studied. The findings were synthesized into the National 
Partnership’s Afterschool Training Toolkit, which was made available to programs 
nationwide via the World Wide Web. Based on the study findings, the National 
Partnership conducted professional development consistently and extensively throughout 
the nation. 


CHAPTER 1: 
INTRODUCTION 


Since the implementation of the No Child Left Behind Act, a greater emphasis has been 
placed on academic development during the afterschool hours. Research has found that 
students’ participation in afterschool programs is beneficial to academic achievement and 
social adjustment (Lauer et al., 2006; Vandell, Reisner, & Pierce, 2007). A longitudinal study 
conducted by Mahoney, Lord, and Carryl (2005) found that students who participated in 
afterschool programs had significantly higher reading achievement and were rated by 
teachers as having a greater expectancy of success than students who did not participate in 


afterschool programs. 


One mechanism in which afterschool programs influence students’ academic 
achievement is through the provision of homework assistance. An evaluation of 21‘ Century 
Community Learning Center (CCLC) programs conducted by Mathematica Policy Research 
(Dynarski et al., 2003) noted that homework sessions tend to be the most common type of 
academic support both in elementary and middle school programs; yet the quality of support 
is generally low. At the same time, studies have found that homework and tutoring programs 
that rely on untrained or minimally trained volunteers often do little to boost students’ 
academic performance (Fashola, 2002). Instead, research on effective approaches to teaching 
and learning continue to emphasize the need to actively engage students and delve deeply 
into subject matter, providing “opportunities to learn with understanding” in combination 
with “a deep foundation of factual knowledge” (Bransford et al., 2000, p. 16). 


In addition to the challenges around hiring qualified staff, afterschool programs can 
also be plagued with attrition and a low frequency of student participation (Dynarski et al., 
2003; Grossman et al., 2002). Attrition and low attendance are of particular concern because 
research has found that those students who participate in afterschool programs the longest 
(both in terms of frequencies and durations) make the biggest gains (Goldschmidt, Huang, & 
Chinen, 2007; Vandell, Reisner, & Pierce, 2007) and higher rates of participation in 
afterschool programs resulted in higher scores on academic standardized tests in mathematics, 
literacy, and language arts (Huang, Gribbons, Kim, Lee, & Baker, 2000). Similarly, Munoz 
(2002) found a positive correlation between the number of visits to afterschool programs and 
improved school attendance and academic achievement. Thus students who infrequently 


attend the afterschool programs are unlikely to reap academic and social benefits. 


In an effort to identify and incorporate quality practices into existing and future 
afterschool programs, the U.S. Department of Education recognized the need for a large- 
scale project to study high functioning 21° Century Community Learning Center (CCLC) 
programs that would result in the creation of strategies, tools, and professional development 
for afterschool programs. In 2003, through a competitive solicitation process, the National 
Partnership for Quality Afterschool Learning (National Partnership), led by SEDL and in 
partnership with the Mid-continent Research for Education and Learning (McREL), National 
Center for Research on Evaluation, Standards, and Student Testing (CRESST), Northwest 
Regional Educational Laboratory (NWREL), SERVE Center at the University of North 
Carolina at Greensboro, WGBH Educational Foundation, and Institute for Responsive 
Education (IRE)', contracted with the U.S. Department of Education for a 5-year project to 
support program quality among the growing number of 21° CCLCs throughout the United 
States. 


This 5-year project provided strategies, tools, and technical assistance to afterschool 
programs to address two continuing challenges identified via research on afterschool 
programs: (a) ensuring that programs offer high quality, researched-based academic content 
utilizing appropriate methods of teaching and learning, and (b) ensuring that programs are 
able to attract and retain students who participate regularly and thus can benefit from these 


investments. 


There were five major tasks for this project aimed at improving the delivery and quality 
of academic content, teaching, and professional development in afterschool programs. 


¢ Task 1. Identification of afterschool sites across the United States that are 
demonstrating exemplary or promising practices. 


¢ Task 2. Validation of afterschool success in the content areas of literacy, math, 
science, arts, technology, and homework help through data analysis and site visitation. 


¢ Task 3. Product development of tools, models, expertise, and other assistance to 
increase the number of quality afterschool sites across the United States 


¢ Task 4. Provision of technical assistance to promising afterschool sites to help 
them achieve “exemplary” status and support state education agencies in building their 
own capacity for technical assistance, including assisting their grantees in a similar 
technical-assistance-building process. 


¢ Task 5. Partnering with the U.S. Department of Education and state education 
agencies to provide professional development opportunities for afterschool sites in 
adopting high quality practices, specifically in content areas designed to increase 
student achievement and attract high levels of student participation. 


' The IRE left the National Partnership in 2004. 


To complete Tasks 1 and 2, CRESST conducted a descriptive study to document 
practices to be used by the National Partnership to 1) develop resources and professional 
development that address issues relating to the establishment and sustainability of afterschool 
programs, 2) provide models and indicators of promising practices, 3) highlight other 
descriptive information that local sites can access in planning new afterschool programs or 
improve existing ones (e.g., the Promising Practices in Afterschool System and the Harvard 
Family Research Project’s database of afterschool programs), and 4) assist in assessing the 
effectiveness of afterschool programs in general, including attention to performance 
standards, review and meta-analyses of research, and rigorous evaluation methods to identify 


“what works.” 


In Years 1 through 3 of the project, Task 1 was completed after CRESST conducted a 
comprehensive search and selection process involving primary and secondary screenings of 
afterschool programs throughout the United States. These screenings were based on an 
established set of criteria, including evidence of academic performance of attendees, Annual 
Performance Report (APR) data, 21° CCLC Profile and Performance Information Collection 
System (PPICS) data, sample size, and recommendations from regional partners. As a result 
of this process, a total of 53 afterschool programs were identified for study and recruited for 
site visits. For Task 2, National Partnership team members conducted 96 site visits at 53 
afterschool programs. Interviews were conducted with the project director, site coordinators, 
and program instructors. In addition, surveys were administered to site instructors, site 
coordinators, and parents. Based on CRESST’s findings, Task 3 activities included the 
development of toolkits and self-assessments created to provide afterschool programs with 
research-based tools and strategies to improve curriculum implementation in the six content 
areas of study (literacy, math, science, technology, arts, and homework), and to evaluate their 
programs regularly and independently. Products, such as the Afterschool Training Toolkit, 
were made available to programs nationwide via the World Wide Web. For Tasks 4 and 5, 
the National Partnership conducted professional development consistently and extensively 
throughout the nation, and collaborated with the U.S. Department of Education and state 


education agencies to build the technical assistance capacity of 21‘ CCLC program staff. 
Purpose of the Report 


This report provides a synthesis of the qualitative and quantitative findings from the 
field study conducted across the nation and the validation of promising practices across the 


six content areas. The following chapters describe the 13 criteria presented in CRESST’s 


Theoretical Logic Model (see Figure 1) and the study design, methods, and results of the 
validation of promising practices in these content focuses. Chapter 2 discusses the program 
selection process and CRESST’s process in developing the indicator system that guided 
validation. Chapter 3 explains the National Partnership’s data collection processes and 
elaborates on evaluation methodology, procedures, and instruments employed by CRESST in 
the analyses. Chapter 4 describes the internal program structure of the 53 afterschool 
programs in the study sample. Chapter 5 provides the findings on program process, including 
parent involvement, connecting with the community, and relationship building with students. 
Chapter 6 examines program-based content practices in terms of curricular goals, alignment 
with state standards, and links to day-school curricula. This chapter also summarizes findings 
on student enrichment and research-based practices. Chapter 7 describes afterschool program 
evaluation efforts and summarizes the results of program impact. The final chapter highlights 
implications from the study, including common promising afterschool practices and 


conclusions for quality afterschool programming. 


CHAPTER 2: 
STUDY DESIGN 
IDENTIFICATION OF PROMISING AFTERSCHOOL PRACTICES 


This chapter describes the National Partnership’s data collection processes, and 
elaborates on methodology, procedures, and instruments developed by CRESST. Year 1 of 
the study focused on literacy and math content practices, Year 2 on science and arts content 
practices, and Year 3 on technology and homework help practices. An indicator system was 
designed first to guide the instrument development and validation procedures. Then a process 


was developed and implemented to identify and select programs in the study. 
Preparation Process for Development of the Indicator System and Standards 


To develop an indicator system for program identification and validation, CRESST 
drew on a number of sources for information and expertise throughout the process. An 
extensive review of the existing research literature on afterschool programs was conducted, 
investigating the organizational, curricular, and environmental variables that have been 
linked to program quality. CRESST then reviewed publications from organizations across the 
country involved in afterschool program evaluation and support. These organizations 
included: the National Center for Community Education, Promising Practices in After School 
Systems, the National Institute on Out-of-School Time, the Harvard Family Research Project, 
the National Community Education Association, the After School Alliance, the After School 
Corporation, the C.S. Mott Foundation, Learning Point Associates, Manpower 
Demonstration Research Corporation, and the Institute for Educational Science. A review of 
these sources focused on common variables and processes associated with positive 


afterschool program outcomes. 


CRESST also received considerable guidance for indicator development from the 
National Partnership’s content focus teams, which comprised experts in the field related to 
each academic area. The math team was led by McREL, the literacy team by NWREL, the 
science and homework help teams by SERVE, and the arts and technology teams by SEDL. 
Harris Cooper, a National Partnership Steering Committee member who is a recognized 
expert in homework studies, provided expert guidance in homework and technical issues, as 
did Elizabeth Reisner, Priscilla Little, and other members of the Steering Committee. Team 
members convened and provided CRESST with feedback on key curriculum content in their 


respective areas of expertise. 


The Indicator System to Guide the Validation Process 


To design an indicator system that could validate promising and exemplary practices, it 
was necessary to define the essential elements in a quality afterschool program. In general, 
quality afterschool programs were defined as having the ability “to deliver basic 
developmental inputs, which translate into practices and principles at the staff, program, and 
organizational level, and which result in positive outcomes for participants” (Hall, Yohalem, 
Tolman, & Wilson, 2003, p.51). Child development theories such as those by 
Bronfenbrenner (1979), Piaget (1952, 1964), and Vygotsky (1978) helped to provide a better 
understanding of how afterschool programs benefit student learning. These theories describe 
the importance of the social context of learning and how learning or cognitive development 


can be stimulated through interaction with peers and adults. 


After extensive literature review and consultation, three central themes emerged: 
program quality evolves from (a) goal setting and evaluation, (b) curricular quality, and (c) 
program environment. Goal setting and evaluation involves theory and research-based 
programming, program structure, setting desired outcomes, and ongoing evaluation to ensure 
that goals are being met. Determination of curricular quality entails academic collaboration 
(linkage of afterschool programs to the day school), and opportunities for students to practice 
skills and reinforce motivation and engagement. Understanding the program environment 
includes social collaboration (school-staff communication), adequacy of physical and human 
resources (e.g., student-staff ratio, space, and materials), and literature on staff education, 
professional development, and attitudes (for more details see the program review plan in the 


National Partnership’s Year | report). 


Findings from the literature served as the basis for the establishment of 13 indicators of 
success in delivering quality content and maintaining effective functionality in an afterschool 
setting: 


» Setting of clear goals for content area practice (1.e., literacy, math, science, arts, 
technology, and homework help) 


» Alignment of research-based activities to achieve goals 

» Alignment of practice content materials with state standards 

» Links between practice content activities and day-school activities 

» Use of research-based curriculum and teaching strategies 

» Provision of a positive program environment 

» Employment of motivational strategies to engage students in learning 


» Promotion of student engagement (e.g., encouraging meaningful experiences) 


» Effective program management/support/resources (e.g., staff/student ratio, staff 
educational experience, ongoing evaluation) 


» Provision of opportunities for student practice 
» Periodic evaluation to check program effectiveness 
» Periodic assessment to review student progress 


» Resetting goals according to assessment results 


These 13 indicators were built into a logic model (Figure 1) to guide the validation process of 


identifying promising and exemplary practices for this study. 


Program Support 
Management 
Staff /Resources 

Periodical Evaluation for 
Content Continuous Improvement 


Process 


Figure 1. Theoretical logic model. 


Further, the 13 indicators guided CRESST’s development of observation and site visit 
protocols and aided National Partnership team members in the implementation of the 
resulting instruments. The model provided a framework to examine the larger context of the 
afterschool environment and assess the effectiveness and efficiency of content practices at 
participating sites. One of the main objectives of this final report is to highlight the 


components of the theoretical logic model that are most likely to lead to program success. 


Program Identification 


To accomplish Task 1 on this project, CRESST identified 21‘ CCLC programs across 
the United States that demonstrated promising’ practices in six content areas: literacy, math, 
science, arts, technology, and homework help. Programs in all of the content areas were 
selected based upon 1) nominations from state coordinators and experts in the afterschool 
field, 2) Annual Performance Report (APR)/Profile and Performance Information Collection 
System (PPICS) data, and 3) telephone interviews with afterschool program representatives. 
Program selection for literacy and math occurred in Year 1, for science and arts in Year 2, 
and for technology and homework help in Year 3. This section describes the process by 


which the programs were selected for the study sample. 
Program Criteria 


The APR/PPICS were the primary data sources used to identify promising programs.” 
These reports were submitted by the 21st CCLC programs, and were provided to CRESST by 
the U.S. Department of Education. The APRs provided information including program 
objectives, grade levels served, number of students served, student demographics, student 
academic achievement data, hours/days per week the specific content curriculum offered, 
number of staff members in the program, and percentage of credentialed staff members. 
Information from 1,600 grantees was available for the selection process. Based on the 
variation in program types and size with different curricula foci, different criteria for 
selection were established across the distinct content areas. The criteria for each content area 


were as follows: 


Year 1: Literacy and Math: 


Each program should have 100 students or more. 


Each program should have a minimum of 3 years of operation history to ensure that the 
content practice has the capacity to maintain its effects and show some signs of stability. 


Each program should show that at least three of its sites offer the requisite content 
practices, as an indicator of stability and to improve the probability for the future success 
of practice duplication. 


*To be identified as exemplary, programs must have had a quasi-experimental or experimental, external 
evaluation that showed evidence of program impact on student learning. No programs met this criteria; 
therefore, all programs in the study were identified as promising practice sites. 

>In Year 2, the U.S. Department of Education contracted Learning Point to convert the APR into electronic 
versions called PPICS. 


Content practices should be offered a minimum of three times per week in order to allow 
sufficient time for student exposure to practices and materials, and to strengthen dosage 
effects. 


Year 2: Science and Arts: 


Each program should have 50 students or more. 


Each program must show a minimum of 3 years of operation history to ensure that the 
content practice has the capacity to maintain its effects and show some signs of stability. 


Each program must show that at least two of its sites are offering the requisite content 
practices, as an indicator of stability and to improve the probability for the future success 
of practice duplication. 


Content practices must be offered a minimum of two times per week in order to allow 
sufficient time for students to be exposed to the practices and to absorb the materials. 


Content practice at both arts and science programs should have a strong focus on at least 
one of the major disciplines. For arts programs, this could include a focus on dance, 
music, theater, or visual arts. For science programs, this could include a focus on physical 
science, life science, earth and space science, science and technology, science in personal 
and social perspectives, or history and nature of science. 


Year 3: Technology and Homework: 
Each program should have 50 students or more. 


Each program must show a minimum of 2 years of operation history to ensure that the 
content practice has the capacity to maintain its effects and show some signs of stability. 


Each program must show that at least two of its sites are offering the requisite content 
practices, as an indicator of stability and to improve the probability for the future success 
of practice duplication. 


For Technology Only: Each program must show a minimum of 1 year using the 
technology curriculum. 


Most important in the identification and selection process was the evidence of success 
in promoting student learning. Data in the following categories from the 2002 APRs for Year 
1, 2004 APRs and PPICS for Year 2, and 2005 PPICS for Year 3 were used in this process: 


1. Meeting/exceeding goals (as stated in the APR/PPICS). Programs that exceeded 
any of their stated goals were identified. A score was given for meeting, exceeding 
or not meeting their goals. 


10 


2. Teacher survey results. The 10 items constituting the teacher surveys reported in the 
APR/PPICS were reduced to one dimension using factor analysis (each of the items 
correlated with this one dimension with correlations ranging from .71 to .92). Higher 
scores were given indicating more “yes” responses to the teacher survey questions, 
generally implying higher quality. 


3. Academic performance of attendees. Reported academic data came in three forms: (a) 
grade gains, (b) grade levels (above proficiency, at proficiency, etc.), and (c) percentile 
ranks. Since only a small subset of programs reported both percentile ranks and grade 
level gains scores for their attendees, it was necessary to standardize each score by 
converting it into a z score (mean = 0, standard deviation = 1) to make the different 
metrics reasonably comparable. To assure that programs working with low academic 
performers would not be penalized, a matched gains procedure was used. This process 
allowed programs starting with very low achievement, but demonstrating significant 
gains, an equal opportunity to be recognized and selected. Thus, academic scores from 
all program sites were rescaled and measures of percentage gains over the previous 
academic year were created. The scores were then aggregated from the site to the 
program level and weighted by the number of attendees at each site. 


All of the scores derived from the APR/PPICS data were correlated to determine the 
consistency of results. The correlations were surprisingly low across the measures, and this 
necessitated focusing on the top performers by content area (i.e., basing standardized units 
above the mean grade level change or percentile rank change). In this way, even though the 
units of measure were not exactly the same, it was less significant because it was a center’s 


deviation from the mean that was most critical. 


Finally, while the original intent was to select only those afterschool programs 
demonstrating high achievement gains and exceeding their goals, relatively few of the top 
performers exceeded their goals. Thus the original criteria were modified to require that at a 
minimum, selected afterschool programs met all of their stated goals. Further, given that 
academic scores were unlikely to capture the totality of program quality, steps were taken to 
expand the pool beyond those showing the greatest academic gains. For the final sample, all 
programs exceeding one or more of their goals were selected, as long as they did not 
demonstrate any academic declines. In Year 1, using a blinded procedure to validate the site 
selection methodology, an additional 5 math and 5 literacy programs were randomly selected 
and blended into the final pool to serve as a comparison control. 


Secondary Screening 


The initial identification process resulted in the selection of 342 programs (i.e., 47 
literacy and math programs, 157 science and arts programs, and 138 technology and 
homework help programs). Additional screening activities were instituted to narrow the pool 
of 342 potential programs. National Partnership staff (blinded to the control procedures) 


contacted the selected programs by telephone to introduce them to the project and collect 
updated and/or additional program information not available in the APR/PPICS. The 
telephone screening with program representatives, most often project directors, covered three 
primary topics: (a) program background, (b) content focus, and (c) _ self- 
evaluation/assessment methods. In terms of basic background information, the 
representatives were asked how long the site had been in operation, how many sites existed 
within their program, the population(s) served, the number of days and/or hours a week the 
program was in operation, and how they would characterize the relationship, if any, between 
the afterschool program and the day school. They were also asked about program goals, 
content areas covered, curricular links to standards, the nature of program instruction 
(purchased versus self-designed curricula, types of activities provided in the instruction of 
the six content areas, and components of their programs that were particularly noteworthy. 
Screening questions also focused on internal and external program evaluations and evidence 


of specific content impact on student achievement, learning, retention, and attendance. 


The representatives were also asked to provide supporting materials including program 
brochures, sample lesson plans, goals and objectives, curricular mapping documents (i.e., 
linking afterschool content area instruction with state standards or day-school curriculum), 
and any other evidence of effective practice that the programs were willing to share. For the 
21st CCLC programs CRESST initially selected, the two sites identified using the 
APR/PPICS data were discussed with the afterschool program’s project director. The project 
directors were also asked to recommend two of their sites for the study site visits, in the event 


that their program was selected as part of the final sample. 


Program recommendations, the additional information from the secondary screening, 
and the initial identification data were used to narrow the total number of programs down 
from 342 to 10 programs per content area. In addition to the criteria previously mentioned, 
program finalists were also selected based on factors such as: standards and research-based 
curricula and instruction, links to school day, effective integration of content practices into 
afterschool program instruction, staff qualifications, and evidence of program impact. Also, 
to assure promising practices in the general population were not overlooked, an additional 
seven non-21st CCLC programs* were also recommended by state region coordinators and 
experts in the field to be considered. It should be noted here that the 10 random programs 
thrown into the selection pool as control measures, mentioned previously, were screened out 


during the process. Thus validating the screening procedure was effective. 


“Programs not funded by the 21“ Century Community Learning Centers. 
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Final Sample Selection 


CRESST developed a proposed list of pre-identified programs and sent this to all 
National Partnership partners and Steering Committee members for review and suggestions. 
After careful screening and group deliberations, 60 program finalists’ were selected and 
presented to the National Partnerships’ content focus teams and Steering Committee, and the 
U.S. Department of Education for suggestions and feedback. Once the U.S. Department of 
Education approved the list, National Partnership staff contacted the programs to request 
their participation in the study. In total, 53 programs? agreed to participate. 


As detailed in the Methodology section of this report (Chapter 3), two sites within each 
of the selected afterschool programs were scheduled for the site visits. Specific 
considerations were taken to ensure that rural programs were represented in the final 
selections. Table 1 shows the geographical distribution of the 53 final programs selected. As 
shown in the table, there are eight rural programs and 45 urban programs. The representation 
is highest at the mid-south region with ten programs selected, followed by the southwest and 


northeast with nine programs selected in each. 


Table 1 


Geographical Distribution of Afterschool Programs 


Regions 
Rural/Urban 
Mia- Mia- Mia- North 
oe Northwest Southwest iS ah West Southeast ite Northeast Conan 
R U R U R U R U RU R U R U R_ U © Total 

Literacy 1 2 1 1 1 1 2 2 11 
Math 1 2 1 1 1 1 7 
Art 2 2 1 1 1 1 1 9 
Science 1 1 3 1 1 1 1 9 
Technology 3 1 1 2 1 2 10 
Homework 2 1 1 2 1 7 
Total 0 5 0 9 2 8 2 4 1 4 1 4 2 7 0 4 53 


> Ten candidates were proposed from each content area. 

° The program sample consisted of eleven literacy, seven math, nine arts, nine science, ten technology, and 
seven homework help programs. The number of literacy programs was increased to 11 as one program that was 
initially chosen as part of the math sample was more focused on literacy as seen in its site visits. 
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As seen in Table 2, of the 53 programs, 33 (62 percent) were affiliated with school 
districts, and 20 (38 percent) were community-based organizations (CBOs). Within the 20 
community-based programs, 10 were run by large nonprofit organizations such as the 
YMCA, Boys and Girls Club, LA’s BEST, Foundations Inc., The After School Corporation, 
Children’s Aid Society, and others. None of the programs identified in this study were faith- 
based. 


Table 2 


Programs Affiliations by Content 


CBO 
Content area School District CBO Nonprofit 
Literacy 6 2 3 
Math 4 2 1 
Art 6 2 1 
Science 6 2 1 
Technology 6 1 3 
Homework 5 1 1 
Total 33 10 10 


In addition, science, arts, and technology programs each had two programs actively 
collaborating with local universities or colleges. Except for science (both of the science 
programs that collaborated with local universities were district-affiliated), the large nonprofit 
community-based programs were more likely to collaborate with local universities and 
colleges. Some of the math, literacy, and homework programs used local college students as 
assistants and volunteers, but none of them were observed to be actively collaborating with 


universities or colleges. 
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CHAPTER 3: 
METHODOLOGY AND VALIDATION PROCEDURES 


In order to provide information that useful in improving academic content, teaching, 
and professional development in afterschool programs throughout the United States, the 
study focused on instructional quality by identifying and gathering information on programs 
exhibiting promising practices. A multi-method strategy incorporating both quantitative and 
qualitative data analyses—including surveys, site observations, and interviews—was 
employed in data collection. SEDL staff and partners conducted site visits and data collection 
on identified programs from 2004 to 2007. This chapter describes the study participants, the 


measures and protocols used, as well as the data collection procedures. 
Program Sample 


As described in Chapter 2 of this report, all programs were selected for this study using 
a complex process that included the development of selection criteria, a review of 
APR/PPICS and other data sources, and secondary screenings by telephone interview. The 
final sample consisted of 53 programs throughout the United States and 96 sites’ across the 


programs. 


Most of the programs had been in operation for less than 10 years, with nearly all 
programs reporting their years of operation between 3 and 7. Depending on the geographical 
regions of the programs, the populations served across most programs were ethnically 
diverse, most common were Hispanic and African American populations. Others included 
Asian/Pacific Islanders, Portuguese, Dominican, and Whites. While White students had 
higher representation in the rural programs, most urban programs served a majority of 
English Language Learner (ELL) populations, with Spanish being the most common primary 


language. Additionally, all programs mainly served lower income students. 
Measures/Instruments 


Based on the theories and indicators of the theoretical logic model described in Chapter 
2, a more content focused validation model was designed to provide a structural framework 
for instrument development, examination of specific content practices, and data analyses for 


this study (see Figure 2). 


7 On ten site visits, the site visit team found upon arriving at the program that only one site was willing to 
participate or, in fact, a program was so small that only one site was visited. 
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Indicators Design & Process Immediate Outcome Expected Outcome 


Linkage with 
School Parent/Staff 
Perceived 


Change 


Linkage with 
Standard 


Academic 
Attitudes & 
Skills 


Research-based 


Internal Program Engagement 


Structure 


External 
Communication 
& Support 


Student/Adult 


Interaction Interview 
Observation 


Parent 
Satisfaction 


Figure 2. Validation model. 


As shown in the validation model, high quality content practices are expected to show 
strength in their curricular content as indicated by linkage with day school, standards, and 
research-based practices. The internal program structure is indicated by program 
organization, duration and frequency of instruction, staff experience and education, 
governance/leadership, and professional development. The program process is indicated by 
external communication and support, student/staff interaction, and parent satisfaction. The 
main focus of this study was on program content, and examined how program process can 
enhance curricular content and student engagement in math, literacy, science, arts, 


technology and homework (see Appendix A for content specified models). 


16 


Based on the content specified models, CRESST developed the data collection 
instruments. Different versions of the instruments were created to cover the six content areas. 
The following is a brief overview of these qualitative measures: 


¢ Staff Survey: The staff survey was administered to participating afterschool 
program site coordinators and instructors. The survey included questions about content- 
specific curriculum, general instructional practices and activities, and organizational or 
structural characteristics. 


¢ Parent Survey: A survey was distributed to parents of the afterschool students. 
Items focused on perceived program satisfaction, opportunities for parental 
involvement, as well as student impact. 


¢ Teacher Survey: A survey was distributed to day-school teachers whose students 
were also in the afterschool program for Years 2 and 3.° Survey questions asked about 
their knowledge of the afterschool curriculum, as well as perceived changes in student 
attitudes, knowledge, and performance due to participation in the program. 


¢ Interview Protocols: Various forms of the interview protocol were developed for 
project directors, site coordinators and site instructors to specifically address questions 
that were most relevant to the interviewee. The interview questions gathered 
information on the general nature and structure of the afterschool program to better 
identify the qualities of a promising practice program. The interview protocols covered 
a number of areas including general program background information, content area 
instructional strategies and other student-based activities, program organization and 
structure, external communication and support, student-adult interactions, and program 
outcomes and impact on students. 


¢ Observation Protocol: A structured observation protocol, including both content- 
specific and general instruction items, was used to document program and lesson 
implementation as well as program structure and classroom climate. The protocol 
included scales, checklists, and open-ended questions focused on the content and 
quality of afterschool instructional practice. 


¢ Site Summary Report: The site visit team completed a site summary report based 

on a template CRESST provided. The information in these site reports was used to 

supplement and triangulate with the other data sources. 

CRESST conducted a 2-day professional development for the National Partnership staff 
and content focus team consultants who conducted the site visits. The first professional 
development occurred in March 2004 at the UCLA campus and additional development was 
provided via teleconference for new site visit staff later that year. For Years 2 and 3, 
CRESST provided training via teleconference only. The professional development covered: 

» Pre-site visit preparation activities 


» Understanding of site visit organization and scheduling 


* Teacher surveys were not utilized for Year 1 due to scheduling conflicts. 
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» Survey collection 
» Use of interview and observation protocols 
» Collection of archived data 


» Debriefing/completing site visit reports 
Data Collection and Analysis Methods 


Both quantitative and qualitative methods were used to collect and analyze data from 
the sample programs. A site visit plan was developed for each site. Surveys, interviews, and 
observations at each site were completed. CRESST provided a program-level checklist to 


guide the collection of all data and any additional information shared regarding the program. 
Surveys 


All surveys were distributed to the programs prior to the site visits. The programs 
assisted in the administration of the survey instruments, which were collected by the National 
Partnership team at the time of the site visits. Surveys were administered to all parents of 
afterschool students, day-school teachers whose students were also in the afterschool 
program, and afterschool program staff including site coordinators, instructors, program 
workers, playground workers, activity coaches, partner organization staff, program experts, 
counselors, and volunteers. Completed surveys from each program were returned to CRESST 
and results were entered into an SPSS statistical software program. Frequencies and means 
were calculated according to indicator variables (e.g., program structure, linkage to the day 
school, professional development). These results were triangulated with the qualitative data 


to help provide further evidence of promising practices. 


There was substantial variance in the number of surveys administered and response 
rates across the 53 programs. Afterschool staff members responding ranged from 70 to 233, 
parents responding ranged from 154 to 1000, and day-school teachers responding ranged 


from 62 to 150 across the different content areas (see Table 3). 


Descriptive statistics were used to present the frequencies and means of the survey 
responses. Exploratory factor analyses were conducted and constructs were extracted from 
the surveys as well as from the APR/PPICS data on teacher surveys used for the purposes of 
program identification. It is also important to note that some content areas were 
underrepresented or heavily represented due to the varying response rates across the 


programs. 
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Table 3 


Survey Response Rate 


Content area Afterschool staff surveys Parent surveys Day-school teacher surveys 

Math 74 377 NA! 
Literacy 233 1000 NA! 
Arts 82 254 62 

Science 103 496 150 
Technology 70 319 105 
Homework 83 154 107 
Total 645 2600 424 


' Teacher surveys were not administered in Year 1. 


Interviews 


Project directors, site coordinators, instructors and other afterschool program staff 
members’ who volunteered to participate in the study (NV = 338) were interviewed face-to- 
face by at least one National Partnership site visit team member. The interviews lasted 


approximately | hour. Table 4 illustrates the number of interviews conducted per program. 


Table 4 


Number of Interviews by Position 


Project director sen i Instructor Other 

Content area interviews interviews interviews interviews Total 
Literacy 11 16 30 6 63 
Math 7 14 21 5 47 
Arts 9 16 31 15 71 
Science 9 13 23 11 56 
Technology 8 15 26 5 54 
Homework 10 14 19 4 47 
Total 54 88 150 46 338 


Interviews were audio taped to ensure data accuracy, although participants were given 
the option to decline having their interviews recorded, none did. Interview questions were 


focused on topics related to (a) professional background and/or professional development 


” Including project managers, academic and community liaisons, community partnership managers, school day 
principals, tutors, etc. 


and experience with afterschool programs; (b) program content knowledge, curriculum, and 
instructional methodology specific to instructional practices in the six content areas; (c) 
internal site structure and organizational characteristics of the afterschool programs; (d) 
external instructional communication and support; and (e) program evaluation and areas of 
perceived program impact. The interview protocol varied slightly depending on the 
interviewee’s position. For example, project directors were not asked questions specific to 
curriculum implementation or instructional strategies, while instructors were. The site visit 
team also had the option of conducting additional interviews depending on the specific 


program or site, such as with the day-school principal or other district staff. 


Once the taped interviews were completed and transcribed, CRESST researchers 
entered the data into the Atlas.ti statistical software package to code the interviews. Codes 
and sub-codes were then created using an inductive approach to analyze the data. A general 
review of the transcripts informed the development of an initial code set that reflected salient 
concepts and common responses across programs, sites, and respondents. Researchers used 
the initial code set to qualitatively analyze a small subset of interviews. Upon completion of 
these tasks, researchers condensed initial codes and developed additional codes to better 


reflect the data. Coding reliability was attained through researcher consensus. 
Observations 


On average, National Partnership site visit team members observed two instructors per 
site (see Table 5). Site visit teams prepared site summary reports based on a template 
developed by CRESST to provide relevant information observed during their site visit 
experiences. These reports, additional field notes, and any supplemental materials collected 


were sent to CRESST and used to further triangulate the findings. 


Table 5 


Number of Site Visits and Observations 


Content area Total # of programs Total # of site visits Total # of observations 
Literacy 11 20 38 
Math 7 14 30 
Arts 9 16 48 
Science 9 18 32 
Technology 10 14 31 
Homework 7 14 25 
Total 53 96 104 


20 


CHAPTER 4: 
INTERNAL PROGRAM STRUCTURE 


This chapter provides descriptions on the internal program structure of the 53 programs 
in this study, including details about program organization, program governance and 
leadership, duration and frequency of instruction, staff education and experience, 
professional development opportunities, and program resources. Survey and interview data 
were used to capture commonalities and differences among the 53 programs, and to extract 


practices that appeared to align with quality afterschool programming. 
Program Organization 


Across all program content areas, afterschool staff members were asked to rate their 
program’s organization on a scale of | to 5, 1 indicating “Strongly Disagree” and 5 indicating 
“Strongly Agree.” In general, the staff across all 53 programs offered high ratings for topics 
related to program organization. They expressed high levels of agreement with the following 
statements (mean level of agreement above 4.0): students understand the standards for 
behavior; programs address students’ behavioral issues promptly; and program has adequate 
materials and resources. Survey results also indicated that staff members felt they had a voice 
in curriculum development and received adequate support from their supervisors. Fewer staff 
members responded they met with each other regularly to discuss the curriculum. Interview 
data further revealed that almost all programs conducted biweekly or monthly staff meetings, 


but meetings specifically focused on curriculum content were conducted less often. 


pr capt ani ey Deterrence Staff merrbersrreet grocery eee ae ee 
foe een wiheach ctherreguaty 
a aa sudetteanng = todscussthecuncuim ean gta 
netucton 


Figure 3. Mean scores on program organization survey responses. 
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Student Discipline 


In addition to asking program staff about their students understanding of afterschool 
standards for their behavior (mean level of agreement = 4.4), they and other study 
participants noted minimal disciplinary issues for the 53 programs. Moreover, staff members, 
on average, strongly concurred with the statement that their site addressed student behavioral 
issues in a timely and consistent manner (mean level of agreement = 4.5). Site summary 
reports supported these findings, with site visit teams observing that the overall classroom 
climate in these programs was positive, staff-student relations were nurturing, staff and 


student expectations for success were high, and disciplinary concerns were minimal. 


The degree of specificity described by programs regarding their approach to discipline 
varied somewhat. Some programs provided specific examples of behavior problems and how 
they would be addressed, while others provided fairly general descriptions of their 
disciplinary policy. Programs also varied in their level of adherence to the day-school 
discipline policy. Several programs adopted the same disciplinary system used by the day 
schools they served. Others seemed to believe that afterschool discipline policies should be 
slightly more lenient than the day-school policies, given the students’ energy level and 
attention span after a full day of school. For instance, one afterschool art instructor stated that 
she “encountered less behavioral problems than day-school teachers because her students 


responded better to the lenient atmosphere of the afterschool program.” 
Program Governance/Leadership 


Afterschool programs are often characterized by the degree to which they adhere to a 
specific decision-making structure. Generally speaking, those with a centralized leadership 
style reserve the majority of the decision-making authority for upper management, while 
programs with a decentralized structure are more democratic, allowing and sometimes 
encouraging participation by a wider circle of stakeholders. Although both leadership styles 
can be productive, an indicator of high quality afterschool practice involves shared leadership 


and decision making. 


In the 53 programs in the sample, triangulation across survey, interview, and 
observation data regarding decision-making processes was consistently aligned. The majority 
of both site coordinators and instructors at all programs indicated in their survey responses 
that afterschool instructors had an active voice in decisions about curriculum and instruction 
(mean level of agreement = 4.4), and took active roles in program leadership and decision- 


making (mean level of agreement = 4.1). 
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The amount of ownership instructors felt toward decision making seemed to be related 
to the kind of program culture and expectations conveyed to them by site coordinators and 
project directors. Some site coordinators appeared to be stricter than others in maintaining 
their authority over the decisions made at their sites. But in the majority of cases, site 
coordinators and project directors acknowledged the importance of their instructors to the 
day-to-day operations, especially when they considered the instructors’ familiarity with the 
students and their needs. Interviewees at most programs reported that instructors’ input was 
highly valued and frequently determined the end result, as instructors were considered the 
“experts” in their content area. An instructor of an arts program explained, 

My experience has been that the afterschool teachers propose something that we’d like to 

do with the kids after school, and [the director] then just talks to us about what our plans 

are. We kind of report to her in terms of lesson plans and how we incorporate standards 


and benchmarks, but a lot of freedom is really given to us. We teach what we’re 
comfortable teaching and what we’re passionate about. 


One site coordinator expressed appreciation for the staff by stating, “These are 
professional adults . . . and they are the best ones to implement the curriculum, because they 
see. They’re with the students, right there with them, and they know what their levels are and 


what their abilities are.” 


In the case of programs where decision making was characterized as decidedly 
decentralized or “democratic,” administrators consistently described the value of staffs 
content-specific skills and expertise, and as a result, curriculum development was more of a 
group process in which staff members were given a great amount of leeway in designing 
instruction. A project director at a technology program clarified that the latitude his staff had 
was evenly balanced by a strong level of accountability for their curricular choices. 

Obviously, we try to be more democratic. ... So one of the things we try to do here, how 

we want to make this a great place to work, is in finding great people, then giving the 

people the power and leeway and the accountability, but also the freedom to do what they 

think works best, and trusting them... . Every quarter they have to come back to us and 

tell us how they’re doing. They report back as to what is going on at their. . . programs. 


In terms of actually decision making and setting goals and deciding what we’re going to 
do, that’s much more of a bottom-up process. 


Overall, data across programs indicated that whether decision-making regarding 
curricula design followed a top-down or a more collaborative model, it was very often guided 
by levels of expertise among staff members. 
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Adequacy of Space and Resources 


Program staff members were also asked about their needs in terms of resources (i.e., 
supplies, staff, space, etc.) in the afterschool program. Consistently across programs, site 
coordinators and program staff commonly mentioned a need for additional space. Many 
programs seemed to rely on access to common space, such as an auditorium or a classroom 
shared with day-school teachers, which often caused logistical problems and sometimes 
prevented planned activities from taking place. Furthermore, some programs expressed 
difficulty with not having consistent access to classrooms. A site coordinator illuminated the 
problem, “I would say physical space would be definitely a big thing with our program . . . 
that is probably one of the hardest things to work with just because every 6 weeks we are 
readjusting the classroom to new classroom seating charts, new areas in which the students 
can and cannot go.” Additionally, several interviewees commented on the need for a separate 


office and storage space. 


According to most programs, another scarce resource was access to technology and 
particularly computers. Although all the technology programs in this study indicated they had 
sufficient computers and tools to work with (it should be noted here that many technology 
programs were funded by, had a partnership with, or received support from technology 
companies such as Dell, Apple, etc.), other programs were frustrated by old equipment and a 
lack of current software. While survey results indicated most of the programs had sufficient 
materials and supplies to work with, a few interviewees also requested more access to 
textbooks, literature, and general supplies. Site coordinators at a few programs also indicated 
that they would utilize additional funds they raised through fund-raising and other activities 
not only to purchase supplies, but also to hire staff with expertise in the field of education or 
social work due to program popularity and growth. 


Opportunities for Practice 


The internal structure of an afterschool program is also impacted by the instruction 
provided to its students. Miller (2003) states that successful afterschool programs provide 
activities that enable students to gain knowledge and to practice knowledge learned in school, 
as well as opportunities to reflect, make decisions, and solve problems. Similarly, the U.S. 
Department of Education and U.S. Department of Justice (2000) found that students need the 
opportunity to practice and develop their literacy skills through intelligent discussions with 


adults and peers, storytelling, reading and listening, games and other activities. 
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Duration and Frequency of Instruction 


The duration and frequency of instruction offered at the identified programs was also 
studied to understand the instruction time and opportunities to practice students were 
receiving as part of the program’s organization. Most of the 53 programs operated 5 days per 
week. The majority of programs reported offering three or more activities each day. Most 
included some kind of homework help or tutoring as part of their programs, but other 
activities offered ranged from academics (e.g., math, literacy, writing, science) to non- 
academic enrichment (e.g., arts and crafts, cooking, gardening, health and nutrition, cultural 


activities, computers) and recreation (e.g., sports, dance, drill team, outdoor games). 


The duration of content specific instruction reported by interviewees varied from 45 
minutes to a little over 1 hour per session, as seen in Table 6. All programs offered their 
content-specific instruction at least two times a week. Homework programs reported having 
the least number of minutes (45 minutes) dedicated to homework assistance on a daily basis. 
However, these programs also offered homework help more frequently (4 days a week) than 
the other content areas. Technology programs reported having the most number of minutes 
(105 minutes) dedicated to activities using or teaching technology and offered technology 
instruction an average of nearly 4 times per week. Technology instruction was unique from 
the other content areas since, while it was often taught as a discreet course, it was also 
consistently integrated into other content areas. For example, the afterschool staff reported 
using technology on a regular basis for academic instruction, as well as during enrichment 
and recreation activities. Students were receiving additional exposure to the academic 
instruction and were given time to practice their skills. Site summary reports across the 
programs also indicated that site visit teams observed that students were mostly engaged and 


attentive, and enjoyed the activities. 


Table 6 


Duration and Frequency of Instruction by Content Area 


Content area Average duration of daily instruction Number of days offered per week 
Literacy 51 min. 3.20 
Math 66 min. 2.58 
Science 77 min. 2.28 
Art 77 min. 3.72 
Technology 105 min. 3.55 
Homework 45 min. 4.00 
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Staff and Staff Development 


Scott-Little, Hamann, and Jurs (2002) state that successful afterschool programs are 
characterized in part by having well-qualified and well-trained staff. Fashola (2002) notes, 
“Academic subjects taught during the afterschool period require qualified, preferably 
certified, instructors familiar with and who can be held accountable for student outcomes” (p. 
60). Similarly, the U.S. Department of Education and U.S. Department of Justice (2000) 
believe that it is important to provide professional development to staff members to increase 
their ability to develop and implement developmentally appropriate curriculum; improve 
their skills in supporting and encouraging curiosity and exploration; support their function as 
role models; foster their ability to instill a healthy self-image in students; and to more 


generally attract and retain high quality staff members. 


Professional development can also give employees ideas for enrichment and hands-on 
activities, greater expertise in academic subject matter, knowledge in assessing student 
progress, and strategies for the different program components of academics, enrichment, and 
recreation. Staff education, professional development, and attitudes are thus important 


program features in afterschool settings. 
Staff Experience and Education 


Through interviews and surveys, site coordinators and instructors at the 53 programs in 
the study sample were asked about their overall experience with afterschool programs and 
their qualifications. Responses gathered from all the content-specific programs were very 
similar. Of the 150'° program staff members for whom these data were available, the 
majority (43 percent) had experience in afterschool for 3 to 5 years. In general, survey results 
were compatible with the interview data, although interview data tended to indicate slightly 
higher levels of staff experience. The results for instructor experience are displayed in Table 
ce 


'°Number of instructors by content area: Literacy N = 30, Math N = 21, Arts N = 31, Science N = 23, 
Technology N = 26, and Homework N= 19. 
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Table 7 
Staff Experience in Afterschool (N = 150) 


Years experience in afterschool Instructors 
Less than 1 year 13% 
1 to 2 years 15% 
3 to 5 years 43% 
6 to 9 years 14% 
10+ years 9% 
Did not report 6% 


In addition to overall experiences in afterschool programs, program staff members were 
asked about their experiences at their current site. The majority of the staff stayed over 3 
years. Math and literacy program staff members had an average of 3.5 years of experience at 
their sites. Forty-six percent of staff in the science sites, 42 percent in the arts sites, and 38 
percent in the technology sites worked over 4 years at their sites, with 75 percent had been at 
their sites for at least a year. Thirty percent of staff in the homework programs had from 4 to 
7 years experience at their sites and 60 percent had been with there for at least a year. Since 
staff stability is important for relationship building, especially for at-risk students, the length 
of time staff stay in a program is important. These staff can serve as constant figures and 
mentors in their students’ lives and their relationships can provide the basis for students to 
build trust, character, and efficacy, which are all essential elements for good citizenship 
(Huang et al., 2007; Huang et al., 2006). Specific percentages for staff experiences at the 


sites are available in Appendix B. 


Program staff members were also queried about their educational qualifications. As 
seen in Figure 4, of the 78!’ staff members that responded, 52 percent reported some type of 
teaching certification, credentialing, or degrees (e.g., credential, bachelor’s or master’s 
degrees). Programs that were district affiliated responded that most or all of their staff 
members were certified teachers, often at the school where the students also attended 
afterschool. At these programs, some teachers also held additional credentials such as 
English as a Second Language (ESL), literacy, math, or special education specialists. 


Community-based programs generally hired from the community, having few certified 


''Number of instructors by content area: Literacy N = 8, Math N = 11, Arts N = 13, Science N = 17, 
Technology N = 15, and Homework N= 14. 
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teachers with staff members typically community members or college students. These 


findings also support the results from the day-school teacher surveys. 


Staff Education Level 


Currently in 
high school 
1% 


Associate's 
degree 
9% 


Figure 4. Instructor education based on staff survey data. 


The majority of program staff reported having a bachelor’s degree (47 percent), while 
24 percent had a master’s degree. In terms of content-specific areas, nearly 45 percent of the 
math and literacy instructors reported having a bachelor’s degree, and approximately 20 
percent had earned a master’s. Similarly, the majority of arts and science instructors also 
reported having a bachelor’s degree (34 percent and 56 percent, respectively), and 28 percent 
of the arts instructors and 36 percent of the science instructors reported having earned a 
master’s degree. In addition to a bachelor’s or master’s degree, 10 percent of arts instructors 
and 14 percent of science instructors also had obtained a teaching credential. As for 
technology and homework program staff, the majority of technology instructors received an 
associate’s degree (46 percent) or a master’s degree (31 percent), while 23 percent had 
earned a bachelor’s degree. The majority of homework instructors received a bachelor’s 
degree (36 percent) or a master’s degree (28 percent), while 16 percent of the homework 
instructors had also received a teaching credential. Compared to the general afterschool field, 


the educational level of these afterschool staff members is high. 
Types of Professional Development 


The most common types of professional development offered across programs were 
general techniques in working with students, such as classroom management, discipline 
issues, social issues (e.g., cooperation or bullying); and general teaching strategies. A few 


programs had professional development that was tailored to the specific needs of their 
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student population. For instance, one program provided professional development on gang 
awareness/prevention, and a couple others focused on preparing staff to serve students with 
special behavioral needs such as ADHD, or academic needs related to second-language 
issues. Two science programs offered professional development on child abuse and poverty, 
some math and literacy programs offered trainings on helping students in their specific 
content areas, and others had staff development for integrating technology and computers. 
Some programs also provided instructors with professional development on how to motivate 


and connect with students. 


Although all interviewees reported having some sort of professional development 
available through their afterschool program, most program staff members received the bulk 
of any formal or semi-formal content-specific professional development through the day 
school or the district. More specifically, the district-affiliated programs generally encouraged 
staff to attend district professional development together with the day-school teachers. 
Among the six content areas, the technology program was the one that offered professional 
development to their staff consistently. One technology program director reported offering 
“extensive, rigorous, and regular technology professional development” exclusively to 
afterschool staff (1.e., not affiliated with the day school or the district). Staff at the majority of 
technology programs supplemented staff development with self-sought or peer-to-peer 


technology professional development. 


When opportunities were available and staff participated in the professional 
development, they generally found it useful. Instructors particularly appreciated professional 
development that was directly applicable in the classroom, relating to teaching strategies, 
innovative approaches and classroom management. As one program staff explained, “That’s 
the best part—that we can actually incorporate it. You don’t just learn it to say you know it, 


but to actually do it with students.” 


Afterschool staff responses also indicated that professional development aided in 
practical knowledge that led to promotion. Some characterized professional development as a 
means of realizing long-term professional goals, as illustrated by one program staff, 

It has definitely been one of the most beneficial things that we have. I believe that it has 

helped our staff members grow as far as within the program and for their own 


development. Most of the staff members that I’ve worked with or that I’ve known usually 
want to become teachers and so this is a great stepping-stone for them. 
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Staff Participation in Professional Development 


Although opportunities of professional development were offered to the program staff, 
they did not necessarily participate in it (see Tables 8 and 9). Fifty-seven percent of the staff 
reported their programs offered professional development from two to four times a year; 
however, only 26 percent participated this many times a year. Sixty percent of the staff 
reported that they never participated in the professional development offered through their 
programs, and 14 percent reported that they participated once. When the data was analyzed 
by content area, the highest participation rate was from the technology staff; 50 percent 
indicated that they had participated in professional development two or more times a year 
while 42 percent responded that they had never participated in professional development for 


technology instruction. 


Table 8 


Professional Development (offered by site) 


Frequency Percent 
Never 18% 
Once 12% 
2-3 times 22% 
4 times or more 35% 
Don’t Know 13% 


Table 9 


Professional Development (participated by staff) 


Frequency Percent 
Never 60% 
Once 14% 
2-3 times 17% 
4 times or more 9% 
Don’t Know 0% 


Notably, project directors and site coordinators seemed to report higher frequencies of 
participation in professional development activities than the program staff. Several 
explanations were gleaned from the data. First, some project directors were not clear in 
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differentiating the professional development that was offered to site coordinators and other 
members of upper management, as opposed to program staff at sites. Second, it seemed that 
program staff had a more rigid definition of what counted as formal professional 
development, whereas many site coordinators counted weekly staff meetings as professional 
development. At the same time, it also appeared that site coordinators and administrative 
staff, often district level, generally received the majority of the off-site professional 
developments. 


Variations in Professional Development According to Content Areas 


The types of professional development offered at the afterschool programs also varied 
according to their content focus. Most professional development opportunities offered were 
on helping students with literacy and math, applying state standards to the curriculum, and 
connecting with the day school. Fewer opportunities were offered on working with ELLs, 
self-evaluation, and assessment. 


Math and literacy program staff cited similar types of professional development offered 
at their afterschool programs. Figure 5 shows, similar to the interview findings, that a higher 
percentage of site coordinators in literacy and math programs reported receiving professional 
development in most categories (other than working with ELLs). Instructional staff in these 
programs reported receiving less professional development in program and self-evaluation. 


Similar trends were observed for science. 
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Figure 5. Types of professional development offered to math and literacy programs. 


Professional development specific to the arts ranged from an artists’ retreat where 


instructors could share their work and discuss best practices, to dealing with conflict 


31 


resolution and how it relates to the arts, to staff development on specific topics such as 
music, dance or paper-making. More specifically, the majority of art instructors (59 percent) 
indicated that they received professional development on helping students with art projects. It 
should be noted here that 5 out of the 9 arts programs had local partnerships with arts studios 


and experts that served as instructors for these programs. 


More than half of the staff interviewed at the nine science programs (69 percent) 
reported that professional development was offered regularly at their sites. The types of 
professional development offered across programs focused mainly on coverage of science 
curriculum and instruction. Specific professional development sessions ranged from week- 
long science workshops to creative ways of exposing children to science through play and 
exploring, including new ways of teaching different areas of science and staff development 


on specific topics such as pollution and the environment, chemistry, and earthquakes. 


The majority of the technology programs reported that implementing or maintaining 
regular professional development sessions was a goal of the program. All interviewees 
reported having some sort of technology-related professional development available through 
their afterschool program. However, it should be noted that at six of the 10 programs, staff 
received the bulk of any formal or semi-formal technology professional development through 
the day school or the district. A good number of interviewees sought professional 
development on their own (e.g., through conferences or online), and many also received 


technology instructions from more knowledgeable colleagues in the afterschool. 


Meanwhile, about 50 percent of the homework program staff reported that no 
professional development was offered to them. Of those who received professional 
development, 28 of the program staff indicated that they received professional development 
on evaluating the program and self-evaluation. Twenty-two percent of the staff responded 
that they received professional development on connecting afterschool learning with day- 
school curriculum, and 20 percent of the staff reported receiving professional development in 
techniques for helping with homework. 


Summary of Internal Program Structure 


The internal program structure of these 53 programs followed a similar pattern. 
Programs were generally well managed, usually under strong leaders who were able to 
articulate a clear program mission and vision statement and goals, and were able to empower 
and motivate the staff to achieve the program objectives. The students appeared to 


understand the codes of conduct for behavior; when necessary the programs addressed 
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students’ behavioral issues promptly. These programs also had adequate materials and 


resources. 


Program staff felt they had a voice in curriculum development and received adequate 
support from their supervisors. In the majority of programs, site coordinators and project 
directors acknowledged the importance of their program staff to the day-to-day operations, 
especially when they considered the program staff's familiarity with the students and their 
needs. This trusting relationship empowered the staff to set high expectations for their 
students and develop efficacy in their instructional activities. The team culture that existed in 
most of the programs indicated that positive relationships and communication among the 
afterschool staff appeared to enhance their ability to expand their roles; and the frequent 
interaction among program staff also appeared to have a positive impact in fostering problem 


solving, thus decreasing the level of assistance sought from the site coordinator. 


As for scheduling, the duration of content specific instruction reported by interviewees 
varied from 45 minutes to 1 hour and 45 minutes per session. All programs offered their 
content-specific instruction at least two times a week. Technology programs reported having 
the most number of minutes (105 minutes) dedicated to activities using or teaching 
technology and offered technology instruction an average of nearly four times per week. 
Technology instruction was unique from the other content areas since, while it was often 
taught as a discreet course, it was also consistently integrated into other content areas. For 
example, afterschool staff reported using technology on a regular basis for instruction in 
academic content areas (e.g., math, literacy, writing, science), as well as non-academic 
enrichment (e.g., arts and crafts, cooking, computers) and recreation (e.g., sports, dance, 
outdoor games). These findings suggested that students were receiving adequate exposure to 
the instructions and were given time to practice their skills. Across the programs, students 


appeared to be mostly engaged attentive, and enjoying the activities. 


On average, the staff in the 53 programs came with 3 to 5 years of afterschool 
experiences and had low turnover rates, with an average of 3.5 years working at their current 
sites. Most of them also came with either a bachelor’s or master’s degree, especially for the 


science, arts, and technology programs. 


In general, all programs offered professional development at least once or twice a year, 
but staff participation was low. Many staff indicated they never or rarely participated in 
professional development offered by their program. Technology programs had the highest 
participation rate, with the majority of technology programs emphasizing the importance of 


following the recent developments in technology. Of the professional development offered, 


33 


all programs described training on student discipline and classroom management. School 
district-related programs tended to provide greater opportunities and varieties for 
professional development, having the afterschool staff participate in the professional 
development together with the day-school teachers. Participation in this type professional 
development was generally higher. When the program staff did participate in professional 
development, they generally reported it was useful, especially when the professional 


development sessions were directly applicable to the afterschool classrooms. 
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CHAPTER 5: 
PROGRAM PROCESS 


This chapter provides an overview of the process for program implementation at the 53 
programs. Descriptive findings on parent participation, connecting with the community, 
staff-student interaction, and student engagement in terms of opportunities for practice, social 


development, and motivational support are included in this chapter. 
Parent Involvement 


Parent involvement is an essential indicator in external connection and support for high 
quality practices in afterschool programs. Site coordinators in the study sample were asked in 
interviews and surveys about their perceptions of parental involvement in the afterschool 
programs. Distinct themes emerged across both survey and interview data for all 53 programs 
with very few exceptions or variations. Most programs shared similar methods of 
disseminating information to parents, as well as means of encouraging their involvement in 
the afterschool programs. However, despite great efforts in these areas, almost all programs 
showed evidence of fairly low parent participation or volunteerism, as well as low 
percentages of formal parent-teacher meetings. Most of the programs did not have a formal 
structure in place for scheduling regular meetings with parents. Parents expressed satisfaction 
with the programs, some mentioned close involvement with their child’s afterschool 


program, while others only received information from the program. 
Sharing Information and Promoting Participation with Parents 


Site coordinators and program staff across all 53 programs consistently offered a broad 
range of means by which they shared program and student progress information with parents. 
Some of the most frequently cited means of communication were: 1) monthly/semimonthly 
newsletters; 2) pre-academic-year orientation; 3) parents’ nights; 4) community outreach 
activities, such as potlucks and student performances; 5) afterschool nights throughout the 
academic year; 6) letters, notes and phone calls home as necessary regarding student 
progress, attendance, or behavioral issues; and 7) scheduled meetings with site coordinators 


or program staff as necessary. 


Survey results indicated parents’ perceptions of program efforts to share information 
and promote participation were generally favorable (see Figure 6). Overall, responses 
indicated that parents felt the programs made an effort to keep them abreast of program rules 
and practices, and to encourage their participation. About 20 percent of parents indicated that 


the program never invited them to volunteer in the afterschool program, but nearly 40 percent 
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indicated that they were invited to volunteer at least once a month. About 35 percent of the 
parents surveyed stated that their children’s afterschool programs provided program 


information in their home language at least once a month. 
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Figure 6. Parent awareness of the afterschool program. 


Interview data indicated that in over half of the programs all parents were encouraged 
to volunteer at their sites in some capacity (e.g., in the classroom, on field trips, chaperoning 
events), and a few sites offered weekend parent workshops (e.g., crafts, cooking, ESL, or 
GED courses). Despite these efforts, actual parent involvement, including volunteerism, was 
extremely low or nonexistent across the programs. Parent survey results, as shown in Figure 
7, confirmed that the frequency of parent participation was low across all programs. Staff 
identified the key reason was that their programs predominantly serve families where both 
parents work and thus are unavailable to attend afterschool activities on a regular basis, if at 
all. As one program staff at a homework program explained, 

A lot of them [parents] are interested to know what’s going on. .. but as far as coming in 

and checking it out, it is a very small percentage. They do get the newsletters and the 


calendars so they do know what we’re up to. Most parents [however] are working at that 
time; so it makes it hard for them. 
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Figure 7. Parent involvement at the 53 programs. 
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More than half of parents reported receiving frequent, once a month or more, 


newsletters from their children’s afterschool programs. However, slightly more than half of 


parents stated that they never helped out at their children’s afterschool programs; and the 


same percentage reported a failure to attend parent workshops or classes offered by the 


program. 


Parent Meetings 


Survey results also suggest that afterschool staff did not meet with parents on a regular 


basis. As shown in Figure 8, afterschool staff across the programs reported infrequent 


meetings with parents. Almost 50 percent of the staff said that they never or rarely had 


formal meetings with the parents, only 19 percent of staff met with parents on a regular basis. 
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Figure 8. Frequencies of parent meetings at the 53 programs. 


Interview responses regarding formal parent meetings with afterschool staff indicated a 
similarly low occurrence. While some programs had a formal parent meeting scheduled at the 
onset of the school year (typically a group orientation rather than a one-on-one meeting), a 
significant number of afterschool staff stated that they rarely, if ever, met formally with 
parents. However, almost all staff in the 53 sites mentioned that they frequently took the 
opportunity to speak and discuss student issues with the parents when they picked up their 


children in the evenings. 
Parent Perception of the Program 


Although formal meetings and involvement with parents were rare, the parents’ 
perceptions of the program were very high (see Figure 9). Ratings were based on a 10-point 
Likert scale (1 being never, 10 being always). Parents indicated they felt welcome to visit 
their children’s afterschool programs at any time, and that there were staff members available 
to speak to them in their home language. They felt comfortable talking about program 
materials, and felt the staff cared about and respected their children. They also reported that 
afterschool staff dealt with their children’s behavioral problems promptly, and that they were 


notified in a timely manner if their children were not paying attention. 
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Figure 9. Parent perception of the program. 


Interestingly, a pattern of progressively increasing parent satisfaction emerged over the 
3 years. It should be noted that only quantitative data that were uniform across all six content 
areas were examined. Averages for each year were calculated and then compared across the 3 
years. As illustrated in Figure 10, it appeared that parents felt progressively more satisfied 
with their children’s afterschool program over the 3 years. There was a small increase from 
Year | to Year 2, and a higher increase in Year 3 on all four ratings. In reading this table, one 
should bear in mind that the content areas under study were different in these 3 years, as were 
the parents surveyed. This heightened satisfaction could be a result of: content focus, time, or 


improvement in afterschool functioning, or any combination of such. 
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Figure 10. Parent’s satisfaction w/ their child’s afterschool program. 


Connecting with the Community 


Staff at many of the 53 programs mentioned ways in which their programs were 
connected with the community. Among the activities most frequently mentioned were 
through service learning, such as making get well cards for or reading to patients in the 
hospitals, taking trips to nursing homes to sing to the senior citizens, recycling or community 
beautification efforts, and cultivating community gardens. One afterschool program worked 
with a local charitable organization, not only to restore a public garden, but also to hike, take 
horticulture classes, go rowing, and swim in the lake. An instructor at another site described a 
local shopping campaign that entailed giving kids a budget and bringing local storeowners 
into the school to set up a mini version of the economy; and another instructor at the same 
site had a pet store owner come to the school and build a mock ecosystem within the building 
to teach kids about biology and conservation. 


Another way in which the community was involved in the afterschool programs was 
through resources. Most staff members reported benefiting from various types of support and 
involvement from the local community. A broad range of community connections that served 
to enhance learning was reported. Benefits from these partnerships were especially valuable 
for arts, science, and technology programs where supplies were costly. For instance some art 
programs benefited from donated materials and supplies, funding, arts-related outings, and 


artists-in-residence programming. 


40 


Many programs also reported getting volunteers as a result of their affiliations with 
local universities and high schools. Other programs got volunteers from community 
organizations such as the Boy Scouts and Girl Scouts, churches, and Boys and Girls Clubs. 
For instance, in homework help programs many of the staff mentioned that volunteers from 
local universities and colleges were able to get involved in tutoring students. Furthermore, 
afterschool program staff also frequently invited science experts from the community to visit 
their program to offer students a more tangible experience of practical, real-world 
applications for science. As one project director explained, 

What makes it unique at [our program] is we have so much community involvement in 

teaching science. . . .We’ve really tried to get experts in the field to come in... . I don’t 


think that there is any program that has more community involvement in teaching 
students science than ours. 


Another site coordinator from an art afterschool program echoed the importance of 
bringing community members into the program: 

That’s the beautiful thing about coordinating the afterschool program. My job is really to 

find these community-based organizations and individuals that that’s their passion. 

That’s what they like to do, whether it’s art or drawing, cartooning, dancing, singing, 


whatever it is. You’re actually bringing in people in the community that really enjoy 
what they do and want to teach that to kids. 


By bringing community members and resources into afterschool programs, students 
were provided with opportunities to explore interests, identify role models, and become more 
immersed in their community. From the varied ways in which the afterschool programs 
enriched their students’ learning experiences in the community, it appears they have a strong 
commitment to providing enrichment activities to strengthen students’ self-concept, 


character, creativity, and community immersion. 
Relationship Building with Students 


In addition to building relationships with the community, afterschool program staff 
members understand the need to focus on building strong relationships with their students. 
Afterschool literature emphasizes the importance of the contribution of high staff 
expectations and positive attitudes toward students to the quality of the program environment 
and student achievement (Hall et al., 2003). Across the 53 programs, the quality of staff- 
student interactions was studied. Field notes and site summary reports indicated that positive 
relationships and interactions between the staff and the students were observed in all of the 


53 programs. 
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Expectations for Students 


Program staff members were asked about their expectations for students’ performance. 
Although none of the programs responded that their expectations were low, they did differ in 
the ways they framed their expectations. Staff expressed that they felt the relationships they 
built with their students helped around expectations for student performance, disciplinary 
issues, and democracy. For example, about one-third of the programs responded that they 
expected their students to be performing at grade level or better; and nearly one-fourth of 
programs mentioned expecting students to do their best in academic performances. Science 
and arts programs distinctly expected their students to gain exposure and engage in science 
and arts experiences, while technology programs expected their students to use technology as 


a means for continuous learning in different life experiences. 
Democracy in the Classroom 


Although most programs were not designed to gather student feedback or give students 
many choices, especially the literacy, math, and homework help programs, staff from these 
programs enhanced students’ sense of democracy by taking their interests into account when 
making decisions on program activities. For example, one program allowed students to offer 
input on where they would like to conduct their service learning projects; others considered 
students’ activity choices. For example, at one site students were given the option between 


reading and doing homework. 


Perhaps due to the context of the different content areas, arts, science, and technology 
programs placed a heavier emphasis on students’ autonomy. As an example, arts program 
staff stated that student inputs were of great value; hence, student interests had a vast impact 
on art curriculum content. Staff felt these kinds of autonomy helped students in taking 


ownership in their learning process and helped to keep them engaged. 
Motivational Support: Engaging the Students 


In addition to positive relationships, motivational support is important in supporting 
students’ academic achievement (Bempechat, Graham, & Jiminez, 1999; Ryan & Grolnick, 
1986). The U.S. Department of Education (1994) states that increases in student academic 
engagement are associated with quality resources, as well as the pressure or incentives for 
students to invest in academic achievement. Miller (2003) states that students may be 
motivated by personal interest in a topic, the desire to please a teacher, peers, parents, and 
other adults; the knowledge that success has long-term rewards; and a desire to increase a 
sense of their own capabilities. Analyses revealed that a variety of motivational strategies 


were implemented at the 53 programs in the study sample. 


42 


Making Learning Diverse, Current, and Fun 


Diversity in activities was one key way the various programs maintained interest 
among their students. The daily schedule generally comprised a combination of homework 
help and tutoring, academics, and some enrichment and/or recreational activity. Most 
programs had short periods of 30 minutes to an hour per activity, and a few used learning or 


activity centers that allowed children to move from one activity to the next at their own pace. 


Programs tried to engage students by offering real world examples that connected the 
curriculum to their lives outside of school. This was particularly salient among science and 
technology programs. Across these programs, hands-on activities with real-world 
connections were the most popular instructional strategies. More than half of the technology 
programs reported engaging students in authentic, real-world, relevant activities; these 
activities frequently led to the development of marketable and applied skills. For example, 
some technology sites trained students to use software programs to produce music or graphic 
designs. Additionally, many science instructors reported inviting science experts to discuss 
real-world applications of science. The math programs also created activities that related to 
everyday life experiences like banking, shopping, and budgeting. As one site coordinator 
stated, “We use things like BMX bikes or other things that interest them.” Overall, most of 
the staff expressed a desire to further engage students by using relevant and real-world 
examples. They also accomplished this goal by referring to local places, familiar activities, 


and known individuals. 


Additionally, programs attempted to make content more relevant by tying it to current 
events. A few sites indicated the use of newspapers and magazines in class, others mentioned 
studying popular culture like hip-hop and movies, some sites studied popular gadgets, and 
several sites used television as the starting point for an educational activity. Overall, popular 
culture references were readily utilized; national and world news events were rarely 
mentioned. A math instructor highlighted her use of current events: 

When we have reading I like to use, especially on Thursdays when we do current events, 

the daily newspaper from town. I usually pick the story, but a lot of times I let them pick 


an interesting one. The last one we did was a big article about the presidential race. So 
we tied that into math. 


The majority of program staff employed unique and innovative strategies to engage 
students in the afterschool setting, placing a particular emphasis on making learning fun. The 
primary reason given for this approach was the need to maintain interest after a long day at 


school. As one instructor aptly stated “Afterschool should simultaneously be fun and 
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informative.” To accomplish this, a number of strategies were used including cross content 
integration, dialogic and cooperative learning, culturally significant programming, special 
consideration for the students’ activity preferences, and the incorporation of enrichment and 
recreational activities. As mentioned previously, providing a diversity of activities, real world 
examples, and currency in learning were also strategies programs employed. Intentional or 
“disguised” learning was also popular; with sports, games, discussions and journal writing 
used to engage learners while pushing learning into the background. A site coordinator 
explained the approach: 

I think that because a lot of our program is disguised learning, a lot of times the kids 

don’t even realize that they are doing math or that they are doing language arts or 

reading. So, in essence, we have them already developing a lot of social skills. During 


our “Why Be Healthy?” time, for example, when they are competing against other 
schools, they are learning team sports and character traits that are appropriate for youth. 


Providing Opportunities for Social Development 


Social development was emphasized in varying degrees at all programs, and positive 
socialization was generally observed among the students. When asked about program goals, 
about half of the staff members mentioned social development, and overall, student-to- 
student interactions at the 53 programs appeared to be positive, nurturing, and respectful. 
Many of the 53 programs mentioned providing additional program curriculum geared toward 
supporting students’ social development, increasing their self-esteem, and creating positive 
self-images. A site coordinator highlighted the way that all activities are related to social 
development: “I don’t think that there’s anything we do that in some way doesn’t impact 


either their image of themselves or how they interact with the world.” 


Many strategies were used to enhance social development; the most commonly 
reported strategies included the use of group activities, multi-age groupings, rewarding pro- 
social behaviors, and encouraging peer collaboration. One of the strategies employed to 
promote social development was through activities such as art, field trips, sports, and 


character-building programs. 


Almost all afterschool programs offered arts activities for their students, including 
poetry, dance, drama, choir, and drawing. One program instituted an innovative program, 
“Fun with Junk,” where kids created art projects out of recyclables. Other sites put on drama, 
dancing, or singing productions for fellow students, teachers, and parents, thus providing 


opportunities for students to work with and collaborate with one other. 
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Field trips were yet another method used to engage learners and provide real world 
links to the afterschool program subjects. More than half of the programs mentioned taking 
students on field trips. Parents were encouraged to volunteer for field trips; hence the benefits 
were threefold: enhancing education by providing concrete experiences, providing social 
development opportunities as students engaged in planning and discussions of the events, and 
furthering parent involvement. Math and literacy programs generally visited museums, zoos, 
libraries, and fire stations; while some took students to local festivities such as pep rallies. 
Art programs, generally, opted for museums and art exhibitions, for instance as one art 
program instructor explained: 

Our program offers a field trip, for example to the Getty [museum], and most of my kids 

will go on that trip. And that’s a really nice time in the sense that we’re introducing the 


arts to them. We visualize, and then they have these field trips that correlate it together 
and reinforce what we’re doing in class. 


Some art programs arranged for a more interactive field trip experience, for instance, one site 
coordinator took students to the local orchestra so the students would pair up with a 
musician. Science and technology programs favored field trips that supplemented their 
curriculum such as science museums and technology centers. For instance, students at one 
science program were taken on a field trip to Lockheed Martin, a multi-national aerospace 


manufacturer and advanced technology company. 


Character-building programs were also offered through many of the 53 programs. The 
site coordinator at a literacy afterschool program described two programs they offered for 
their students: “The girls get Smart Girls, which is also character building, but it deals with 
those life changes during puberty, taking care of your body in terms of hygiene, and what 
does it mean to be a... . in society. Same thing for men, for the boys it's Passport to 
Manhood.” Another site coordinator for a math afterschool program also mentioned a 
program called Character Development, which focused on teaching students values such as 


“honesty, respect, responsibility, and caring.” 
Summary of Program Process 


In summary, the process for program implementation in the 53 programs showed a 
focus on program and staff relationships with parents, the community, and students. Most 
parent and staff communication was informal, often occurring during pick-up time. Written 
communication, such as newsletters, was sometimes used to share information. Despite some 
limited means of communication, parents felt the programs did a satisfactory job in keeping 


them up-to-date with program rules and practices. Parent involvement across the programs 
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was generally low; however, parents felt good that their children were well taken care of and 
disciplinary issues were dealt with efficiently and fairly. 


With some exceptions, community involvement was not particularly strong at these 
programs. For math, literacy, and homework, community involvement generally consisted of 
service learning opportunities, volunteers recruited from the community, and students 
participating in community events such as paper recycling and beach cleaning. Arts, science, 
and technology programs appeared to have more in-depth collaborations with their 
community through sharing resources, both in materials and in expertise e.g., arts residence 
programs assigned artists to the afterschool program for a period of time to work with the 
students; Dell Corporation brought in materials, lesson plans, and technology assistance to 
the afterschool program; and local museums sent scientists to deliver a project with the 


students. Some of the programs also collaborated with local colleges and universities. 


Afterschool staff and student relationships were characterized by warmth and mutual 
respect throughout the 53 programs. Almost all programs wanted the students to improve 
both academically and socially. In building relationships with students, staff in these 
programs had high expectations for their students and ensured the students were aware of 
these expectations. Staff also provided democracy in the classroom so that students had a 
voice and felt empowered and rules were made fairly and observed consistently. Some 
programs also provided character-building programs and all programs provided motivational 
support as well as desired to engage their students through activities. Consequently, minimal 
disciplinary issues were observed or reported. 


Students’ social development was also important in the process of program 
implementation at many sites. Staff employed techniques such as peer collaboration, 
different grouping arrangements, and cooperative learning opportunities to realize this goal. 
According to field notes, most student-to-student relationships observed at sites were 
positive. To facilitate learning, student motivation, and engagement, programs utilized five 
major techniques. First, they tried to make learning fun; the staff accomplished this goal by 
including recreational activities and disguised learning. Second, the staff broke up the day, 
and as a result students enjoyed a diverse set of activities. Third, the staff used real-world and 
relevant examples to teach the curriculum. Fourth, many instructors discussed current events, 
for example some instructors referred to popular culture and events. Finally, about half of 


programs made use of field trips, student performances, and exhibitions to advance learning. 
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CHAPTER 6: 
GOAL SETTING AND CURRICULAR PRACTICES 


This chapter presents the findings focused on goal setting and curricular practices. Four 
thematic areas will be discussed including setting program goals, standards-based 
curriculum, linkage to the day school, and research-based curriculum and instructional 


practices within specific content areas. 
Goal-Oriented Programs 


Through a meta-analysis of the literature RAND conducted, Bodilly and Beckett 
(2002) found that the setting of clear goals and desired outcomes is essential for program 
success. More specifically, the structure and design of afterschool programs should be guided 
by the purposes of the program. All 53 programs appeared to be able to structure their 
curricular design and program practices to facilitate meeting their program goals. For 
example, three of the math programs and four of the literacy programs specifically targeted 
students who were struggling academically. The site coordinator at a math program described 
their program goals as focused on developing students’ academic skills within the math 
content area: 


Our primary goals are to bring the student to grade level. That’s my primary goal and 
covering the basics of addition, subtraction, multiplication and division. 


Programs with achievement goals were frequently district affiliated and generally ran a 
more structured program, stressing the improvement of academic performance, hiring more 
certified teachers, and maintaining a more consistent linkage with the day school (in terms of 
curriculum coordination, communication, and adherence to state standards). The majority of 
the arts programs had a different focus. Their common goal was to provide students, who 
otherwise would have little to no exposure to the arts, with art experiences. Program staff, 
both instructors and administrators alike, frequently noted that while the afterschool curricula 
attended to the need for academic achievement and progress, arts instruction could also serve 
as a means to encourage creativity and discovery through expression—something most of 
their students had very little experience with due to recent school cut backs and financial 
short fall. As one site coordinator explained 

These are kids that have never been exposed to anything—I mean really nothing... . 

[Our goal is to] open the kids’ minds to new things, and to show that they can do it... 


.We wanted a well-balanced, well-rounded program with a lot of different things to offer 
to the children, to open their minds. 
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A number of interviewees further suggested that by using an integrative approach to arts 
instruction in afterschool programs they could potentially impact students who were 


struggling academically and personally. 


Similar to arts, all science staff responded to questions about curricular goals by 
suggesting a primary interest in simply offering positive science experiences to students. As 
one executive director remarked, “[Our] primary goal . . . is to awaken in students a curiosity 
about science and the world around them.” Beyond this basic theme, a smaller percentage 
(slightly more than 20 percent) of interviewees across science programs characterized their 
curricula as mostly focused on academic improvement and achievement. Most of these 
respondents reported that they considered their programs to be an opportunity to extend 
science projects and labs from the day school to afterschool, as was the case of one site 
coordinator who remarked, “We’re growing on what they’re teaching [in the day school], 
trying to enrich those areas.” A few went even further to explain that their principle goal was 
to improve standardized test scores in science by focusing on extending day-school 
instruction into afterschool. One academic coordinator described designing afterschool 
science curricula with a strong academic focus in order to “bring students up to par for 


standardized testing.” 


Technology programs appeared to have their main focus on developing students’ 
applied knowledge. Nearly all of the technology staff interviewed reflected the goal of 
teaching students the mechanics of a broad range of technology skills, in the interest of 
encouraging the students to use those skills to enhance learning in other content areas. 
Further, their goals included teaching the students a technical skill that can be relevant to 


their real lives, both now and in the future. 


Only one technology program mentioned academic achievement as their primary focus. 
In this program, integrating technology into other content areas “to improve reading and 
math scores” was the primary goal, while staff across the other nine programs consistently 
explained that academics were just a piece of the puzzle, and development of skills in 
technology could serve their students on multiple levels. One project director explained that, 
by showing students that learning through technology can be interesting and relevant to their 
lives, the program could positively impact students’ futures far beyond elementary and high 
school experiences. 

Everything that we do in afterschool, we try to make it somehow connected to the real 


world. ... That’s one of our many goals is to make it relevant, that everything we do be 
relevant . . . to perhaps make a connection for them for a reason for staying in school, a 
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reason to go to college, a reason to be on a particular career path; and if technology is 
what fascinates them then that's what we will use. 


Interview data across the seven homework programs suggested that the primary goals 
in afterschool homework help were to complete homework and increase academic 
achievement. Many staff members reported that assisting the students to understand their 
homework as well as increasing motivation to complete homework was important to achieve 
their goals. 

We recognized that the other role we have to play is to get children engaged in the 

learning process. So it’s not just about completing the homework, but it’s about finding 

ways outside of the school to get them interested, excited, feeling confident, and to build 

their self-esteem so that even if they only do a third of the homework that they feel good 


about it, and they want to come back the next day and try a little harder. That’s giving 
them some sense of motivation. 


In aligning activities to achieve their program goals, nearly all interviewees described 
the merits of developing a curriculum that combined academic skill development with 
opportunities to explore, and encouraging students’ social development. One program 
director summed up well in saying, 

Our goal is to help each child to make sure that it’s an afterschool program that’s fun, but 


at the same time it’s enriching, so they grow and learn every day, so they can take home 
some more knowledge. 


Standards-based Curriculum 


Through interviews with site coordinators and program staff, questions were asked 
related to their familiarity with their states’ content standards, the ways in which curriculum 
are linked to the standards, and how coverage occurred. Additionally, program staff members 
were asked to explain which specific standards were covered in a lesson observed during the 
site visits. Overall, interviews revealed that most programs appeared to incorporate state 
standards into their curriculum. This occurred across programs to varying degrees with some 
knowledgeable about, and purposeful in, delivering a standards-based curriculum, while 


others appeared to make it a secondary, or less intentional, goal. 


Most program staff reported they were familiar with the state standards, although levels 
of familiarity seemed to vary. Some program staff and site coordinators stated they had a 
strong knowledge of the standards, often because they were also day-school teachers, or 
because their program built on the day school’s curricular structure. In about half of the 


programs, especially those that were community based, one or more of the interviewees 
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responded that they were not very familiar with the state standards. In most of these cases, 
interviewees further explained that they had some familiarity, or knew where they could find 
the standards. Also, a few respondents remarked that although they did not know the 


standards, they were aware that the afterschool curriculum they employed was tied to them. 


Similarly, a few of the program staff were able to give detailed explanations of how 
standards-based contents were integrated into their afterschool curriculum. One program staff 
spoke of the ways in which she embedded standards into her instruction: 

What we do is we break down the standards and make sure we fine-tune every standard. I 

just make sure that every word [of] what I’m teaching is a standard. I look in my book, 


and if the standard is not taught fully, I go to a different text or resource that will meet 
that standard. 


Other program staff had more general views of standards-based curriculum For example, a 
site coordinator at a math program had a more general idea of how standards were addressed 
by their afterschool curriculum: 


We mainly come up with [curriculum] from general standards of what students should be 
working on, when they are in kindergarten or when they are in first grade. We take that 
information and a combination of what they are working on during the school day. So, 
we get information from the school day teachers whether it’s weekly or if it’s monthly, 
“My students are working on X, Y and Z this month in math.” Then [afterschool] 
teachers create fun games and activities like that during that time. So, they might have a 
bingo game that’s multiplication or some other thing. 


Survey responses supported the qualitative findings. In Year 1, for literacy and math 
programs, the afterschool staff members were asked to rate their familiarity with state 
standards on a scale of | to 10 (see Table 10). In Years 2 and 3, the science, art, and 
technology program staff were asked to rate their familiarity with state standards on a scale 
of 1 to 5 (see Table 11). A higher score indicates greater familiarity with state standards. In 
general, it appears that math site coordinators were the most familiar, and technology staff 
were the least familiar. Since there are no homework content standards, homework programs 


were not included for this item. 
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Table 10 
Mean Rating of Familiarity with State Standards (Scaled from 1 to 10) 


Programs Site coordinators Program staff 
Literacy programs 8.79 8.30 
Math programs 9.5 8.24 


Table 11 
Mean Rating of Familiarity with State Standards (Scaled from 1 to 5) 


Programs Site coordinators Program staff 
Science programs 4.25 4.38 
Art programs N/A' 3.95 
Technology programs N/A' 3.13 


"No survey responses on state standards were received from site coordinator in the art and technology 
programs. 


For the technology programs, the survey results indicated a moderate level of 
familiarity with the technology standards. While most staff in these programs rated their own 
knowledge of technology standards at the weak to moderate level on the survey, the majority 
of staff interviewed described activities or methodology they employed that was clearly 
reflective of national and/or state standards for practice. For example, the majority of 
programs reported that students regularly engaged in activities aligned with the standards- 
based criteria such as: using technology tools for learning, productions, and creative 
exercises; using multiple technology resources to communicate information and ideas; using 


technology for research; and using technology for problem-solving in the real world. 


Afterschool staff members were also asked whether they employed standards-based 
content instruction in their practices. Results revealed that many staff included standard- 
based content in their curriculum. However, within each content area, some standards were 
used more often than others. In the content area of literacy, 96 percent of staff reported using 
lower grade-level standards as represented by “vocabulary development” and “reading 
silently or aloud with fluency.” The upper grade level standards as represented by “applying 
self-correcting strategies to decode text” and “understanding literary techniques” were used 


by a lower percentage of staff (55 and 61 percent, respectively). Similar to literacy, math 
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staff also reported applying lower grade level standards such as “addition and subtraction of 
whole numbers” and “pattern recognition” more often; only 50 percent of the math staff 
reported on applying higher level standards such as “problem solving using equations” and 
“understanding/applying mean, range, and median.” In general, most science instructors 
indicated employing standards-based curriculum. Due to the vast numbers of contents 
available, there was wide variation in subject selections. For example, only 33 percent 
indicated that they worked with “reproduction and heredity” in their instruction, whereas 80 


percent indicated they used the content “understanding of science/technology.” 


Most of the arts staff reported employing standards-based arts curriculum. Ninety-three 
percent noted they applied the standard to “apply art techniques and processes.” The least 
common practice was in the integration of arts with technology, where only 33 percent of the 
arts staff indicated that they “use technology as a creative tool.” For technology, most of the 
program staff indicated they gave students opportunities to “practice responsible behavior 
and use of technology.” The least used standard, which only 43 percent of technology staff 
reported applying, was “research and evaluate the accuracy and bias of electronic 
information.” Once again, grade level played a role in the staff’s selections of content topics, 
with most of the choices leaning towards the lower grade levels. Since there were no 
homework content standards, homework instructors were not included. The details of the 


standard-based content areas and their use are found in Appendix C. 
Linkage to School 


Northwest Regional Educational Laboratory (2003) emphasizes that afterschool 
programs need a strong link to academics and other objectives in the day school to increase 
student achievement. This continuity of learning between the school and afterschool program 
is supported theoretically in the work of Noam, Biancarosa, and Dechausay (2002). These 
researchers posit that the “bridging” of school and afterschool helps to promote more 
meaningful academic learning. Miller (1995) further emphasizes that it is possible for 
students to “increase [their] sense of themselves as learners” and to “transfer positive 
experiences in a school-based afterschool program to more positive feelings about school 
itself’ (p. 46). As seen in this and other literature, the linkage between the day school and 
afterschool program is an important factor in the quality of afterschool practices. For this 
study, the pattern of communications and relations between the day school and afterschool 


program were examined. 


The staff survey inquired how frequently the afterschool staff communicated with the 
day-school teachers. Slightly less than half of the afterschool staff indicated that they 
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communicated with the day-school teachers frequently or regularly, whereas an equal 
number of afterschool staff indicated that they never or rarely communicate with day-school 
staff. About 20 percent of afterschool staff reported that they sometimes communicated with 


day-school teachers (see Figure 11). 


100% - 


90% 
80% 


7TO% 
60% 


50% 


40% 


Percentage of Instructors 


30% 


20% 


10% 


O% 


Never Rarely Sometimes Frequently Regularly 


Figure 11. Frequency of communication between afterschool program and day-school staff. 


Since a critical component of effective afterschool programming is the coordination of 
program content to the day-school curriculum, the interviews conducted with the afterschool 
staff probed deeper into the connections between day school and afterschool. A number of 
distinct themes emerged from the qualitative analyses. With a few exceptions, most 
afterschool staff reported a more casual form of communication with the day-school teachers, 
typically on an “as-needed” basis in regards to individual student progress and/or behavioral 
issues. Communications with the day-school teachers occurred mostly in forms of brief, 


informal contacts such as email or casual drop-in conversations. 


A few programs described a more intentional approach. For example, overlap in day- 
school and afterschool staff was mentioned as a strategy to allow more collaboration and 
communication between day school and afterschool. In some district-affiliated programs, 
they referred to this process as a “seamless approach.” In other instances, a YMCA program 
made use of agendas, planners, and homework logs to communicate with day-school teachers 
relating to homework assignments, and coordination of curricular contents. Overall, the key 
topics on most of the communication between day school and afterschool were homework 
related, indicating homework was a crucial component of most academic activities in 


afterschool and a means of connection between the day-school and afterschool teachers. 
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An interesting exception was the science afterschool programs, where the majority of 
site coordinators (90 percent) and staff (93 percent), reported that specific staff members 
were assigned or responsible for communication with the day school. At a closer look, six of 
the nine science programs were district affiliated, with two of the other three community- 
based programs indicating strong ties with the day school. Staff at one afterschool program 
described a science program that was clearly the result of a collaborative effort between day- 
school and afterschool instructors, designed around and coordinated with the district’s 
science curriculum, grounded in state and/or national science standards, and supplemented 
with purchased science kit materials. Science lessons revolved around a fish hatchery theme, 
developed primarily during day-school instruction, and maintained and studied throughout 
the year in both the day school and the afterschool program. According to interviewees, this 
concept provided a focus that gave students a beneficial sense of continuity between day 
school and afterschool science instruction. One site coordinator explained, 

What’s really neat about this is that it was a collaborative project. . . . We worked 

together. Not just one school building, but we got together and partnered with another 

building and we made sure that we had representatives from across the grade levels. So 
we’ve got pre-K through fifth grade and then specialists represented, and we designed the 
curriculum. ... We went ahead and tied this curriculum to the national science standards, 
which will make it very easy for our school day teachers to incorporate this program into 


the school day, because it’s electronic. All of their lesson plans are ready to go for this 
program, and it will all be tied to the national standards. 


Some general commonalities were also found in terms of adjusting practice according 
to student need. For example, most district-affiliated programs reported including input from 
their day-school teachers when fine-tuning their curriculum. Other programs also solicited 
input from parents and students; two programs actually employed an education coordinator in 


their programs to manage coordination with the day school in curriculum development. 
Research-based Curriculum and Instructional Practice 


Afterschool program instructors were also asked about the use of research-based 
practices and teaching strategies, including curriculum development and practices in the six 
content areas. Most afterschool programs indicated that they designed the curriculum, rather 
than rely on pre-designed or purchased curriculum. At one arts program, decisions about the 
arts curriculum were clearly a collaborative effort in terms of both afterschool teamwork and 
working closely with community partners and the day school. This program’s approach 


appeared to be integrative in many respects, but most evidently with regard to arts curriculum 
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development. The arts community partnership director at the afterschool program explained 
the process in detail: 
[Our arts program is] self-designed. We don’t purchase any programs. ... We work with 
a variety of artists. .. . We are really responsive to the community. . . . We create 
curriculum that addresses that. .. .The artists ultimately are the ones that carry that out. . . 
. We talk together with them. . . . we meet with the principals and different 
representatives before we do any scheduling for the session, to talk about the curriculum 
areas... . The artists do the lesson plan of the actual lesson. . . . Then we look at each 
session and what an overarching curriculum area might be .... Then we work with our 
artists to come up with a variety of ways to address those topic areas. 


This approach to curriculum design was also echoed by a site coordinator at a science 
afterschool program: “[The curriculum is] self-designed—nothing is cookie cutter here... . 
Our teachers design everything that we do here”; while another site coordinator stated simply 
that the program staff “come up with their own curriculum and they make sure it ties with the 
[state] standards.” 


On the other hand, some programs did indicate using curriculum from the day school or 
that were pre-designed. The project director at another science afterschool program indicated 
that they relied primarily on the day-school curriculum: 

First of all, we use what is being taught in the school day. We take whatever curriculum 

they’re using and enhance that. The way that we do that is we actually use our regular 

day teachers as our afterschool teachers. The curriculum is already there. We use what is 

on their lesson plans. We take from that and work with that as far as what we do in the 

academics. We also advance into that by having projects and programs with research 

based things, such as Design It, which is a science project. We use their curriculum for 

our science activities, Wide Span for reading and math. 


Similarly, a site coordinator at a literacy program remarked that the curriculum used 
was pre-designed and straightforward: “It's very recipe-like. It tells them exactly what 
questions to ask prior to the book, the activity suggestions. It's very laid out for them, so it's 
very user-friendly.” Overall, while some programs chose to use pre-designed curriculum and 
others preferred to design their own, all programs had consensus that they want to employ 
curriculum that would support students’ academic competence, while making learning fun 


and less rigid than in the school day. 


To implement the various curricula, some of the common practices across the six 
contents included scaffolding techniques that appeared to be correlated with grouping 


strategies, and the “real world” approach that was used to encourage student learning by 
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infusing the learning process with cultural and social experiences that were relevant to their 
lives. Other common practices in science, arts, and technology were described as a “hands- 
on” approach; active participation; “learning through doing”; students making connections on 
their own; and the constant shifting from guided work, to group work, to pairs work, then to 


self-paced work, and back once more to guided work at a higher level. 


In general, survey data suggested that afterschool staff employed a wide range of 
research-based practices to support and facilitate their students’ learning. The analysis of the 
survey data revealed that there were variations based on the content focus. For example, 
technology programs reported the most frequent use of research-based practices whereas 
literacy programs reported the lowest frequency. Furthermore, technology, science, arts, and 
homework programs appeared to be more focused on developing higher order thinking skills, 
whereas literacy and math programs were more focused on direct teaching. For instance, one 
technology instructor talked about teaching her students to problem-solve: “You have to 
know how to fix it. How do you solve this problem? The engineer is sitting in that seat now, 
and something goes wrong, you’ve got to troubleshoot. That’s part of life with anything. 
Troubleshooting and problem solving skills are needed.” Specifically, in science content, the 
most frequent practices were (a) describing scientific procedures to students, (b) using tools 
to gather and analyze data, (c) designing a scientific investigation, (d) conduct simple 
experiments, and (e) using evidence to predict and explain. All these strategies were designed 
to promote applicable techniques and higher order thinking skills and were practiced fairly 


evenly across all science programs. 


On the other hand, math and literacy programs were more focused on direct instruction. 
For example, math staff reported the use of mathematical tools, asking students to solve 
world problems, and providing instructions to solve math problems most frequently. The less 
frequently practiced instructional strategy appeared to be in the arena of developing students’ 
higher order thinking skills such as providing unstructured opportunities for students to 
investigate and explore, and having students write justifications for their work. The least 
practiced strategy in math and literacy programs was having students actively participate in 
hypothesis testing. In many of the math and literacy programs, program staff mentioned 
teaching knowledge facts through direct instruction. One math staff commented, “So we have 
pretty much tried to stay with just basic facts for them to learn math facts.” Another literacy 
staff talked about literacy instruction: “In particular I work with two students that their 
parents requested them to stay here with the afterschool program because they don’t have the 
support at home to read and learn vocabulary with their parents. So I try to make a point to 


read and study words with them.” These trends may have been the result of differences in 
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content characteristics, and/or a process of program maturation (since math and literacy were 
visited in Year 1, the afterschool program as a field has gone through rapid development and 
maturation). The frequencies of the usage of specific instructional practices under each 


content area can be found in Appendix D. 


Another strategy was the implementation of cross content integration. Over the six 
content areas, the curriculum for technology was the most integrative; these programs 
incorporated technology learning into many of the afterschool activities across academic 
content areas including science, literacy, math, and arts. One site coordinator commented on 
the integration of technology into the afterschool curriculum: “In most all of our subject areas 
technology is incorporated somehow and integrated into it.” She further expanded: 

I talked about the language arts. The math you could get into how many beats per minute 

for a song. You have to do counting. When someone is doing the background music you 

have to figure out how many counts, how many beats per minute or per stand that they’re 

doing it on. So there’s a lot of math involved with that. Science, I’m sure there’s a 

science, but I can’t pick it off the top of my head. Art, we actually have one of the 

teachers from Henninger’s. She incorporates a lot of art, and especially when they do 


famous groups. They might work on a comic book or a stage set, so they do a lot of 
drawing and things like that or even computer parts. 


Homework programs also had a broader conception of instruction that focused not only 
on homework assistance and completion, but also on developing study skills, work habits, 
and organization skills. The four skills most staff mentioned enforcing were: development of 
time management skills, development of skills in using reference material, development of 
note taking skills, and strategies for test preparation. For instance, one homework staff 
member commented on modeling note-taking for her students: “Ill give notes to try to 
explain to the kids this is how you should take notes when I’m doing it. You don’t have to 
write everything down, but then there are some teachers that want you to make sure you 
write everything down.” Another staff member mentioned teaching study skills as an 
important component of afterschool homework instruction: “Teachers sort of become experts 
at homework at some point. I do work with my students on study skills. It’s really what I 
focus on. It doesn’t matter if it’s homework or if it’s class work or if it’s just some 


extracurricular activity that they’re working on, but I really work on study skills with them.” 
Summary of Goal Setting and Curricular Practices 


In general, the afterschool programs adhered to the structure of goal-oriented programs 
as reviewed in the literature. Almost all programs under study had set clear program goals 


and objectives, and designed strategic plans to achieve their goals. For example, some of the 
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math, literacy, and homework programs had improvement on academic achievement as their 
program goal. These programs hired credential teachers as instructors, focused their 
curriculum and teaching strategies to enhance student academic achievement and academic 
skill building. 


Math and literacy programs tended to focus more on improving basic skills whereas 
science, arts, and technology programs had goals of exposing and engaging students to their 
specific contents. The science, arts, and technology programs hired and/or collaborated with 
experts in their field and emphasized exposing students to a variety of experiences, providing 
students with enrichments and personal experiences. The staff in these programs tended to 
focus on developing the higher-order thinking skills such as gathering data, analyzing data, 
and reporting/presenting the findings. Homework help programs’ intent was to provide more 
skill building to their students through improving students’ organization, time management, 


and test preparation skills in addition to supervising and monitoring homework. 


At the same time, all 53 programs had broad knowledge of national and state standards 
in their content areas, and alignment with standards was obvious at most sites. However, 
although most programs were employing standard-based curriculum, some of the program 
staff were not well versed in expressing links between standards and their curriculum. As a 
point of interest, while almost all programs adhered to standard-based curriculum, most of 
the programs, regardless of grade levels, tended to focus more on the basic standards rather 
than the more advanced standards in their content area. This may be a reflection of the 
demographics of the student population since all the programs were working with low- 


performing schools, and perhaps populated with more primary grade students. 


Some of the common research-based practices across the programs included 1) a real- 
world approach that was used to encourage students’ learning by infusing the learning 
process with cultural and social experiences that were relevant to their lives; 2) a hands-on 
approach, particularly in science, arts, and technology; 3) giving students the freedom to test 
and expand their own knowledge through doing; 4) emphasizing students “learn through 
doing” and make connections on their own; and 5) constantly shifting from guided work to 
group work to pairs work, then to self-paced work and back once more to guided work at a 


higher level. 


Although all programs appeared to maintain positive relations with the day school, 
most of the linkages were in casual formats and communication with day-school teachers 
often happened on an as needed basis. On these occasions, homework was usually the key 


topic. Some programs, especially the district run programs that usually constituted shared 
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staff, had better communication systems in place A few district run programs emphasized a 
seamless approach where afterschool program could be considered as an extension of the day 
school. A few community-based programs also made specific efforts to maintain open 
channels between the school and afterschool. In one program, a key person was designated to 
liaison between the day school and afterschool. In another program, a system of homework 


logs, checklists, and meeting dates was set to ensure frequent correspondence. 
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CHAPTER 7: 
EVALUATIVE STRUCTURES AND PROGRAM IMPACT 


Program Evaluation 


Nationally and across a number of states, there is a growing emphasis on the evaluation 
of afterschool programs. Those funding afterschool programs want to know that their 
investment is making a difference and are encouraging programs to engage in continuous 


improvement, making assessment and evaluation an integral part of every program. 


There are typically two types of evaluation that apply to afterschool programming: 
internal and external. Internal evaluation is a process of quality review undertaken within an 
institution for its own ends. External evaluation is the appraisal process performed by an 
agency or individuals not directly involved in or responsible for the program or activities 
evaluated. Interview and survey responses across the 53 participating programs in this study 
indicated that even though rigorous examinations of data were rare, all of the afterschool 
programs appeared to be conducting varying degrees of internal and external evaluations of 
their programs. Overall, project directors seemed to be the most knowledgeable about 
evaluation, particularly external evaluations. Site coordinators were usually familiar with 
their individual sites’ internal evaluations and quality assurance procedures. Program staff 
members were most familiar with informal student assessment procedures and tools utilized 


to monitor student progress at the classroom level. 
External Evaluation 


Responses from interviewees suggested that many of the programs were evaluated 
externally, sometimes by an evaluation organization experienced in program evaluation. 
Most interviewees consistently indicated that their evaluations were of the entire program. 
The methodologies utilized in these evaluations typically included pre-post testing or 
classroom evaluations, comparison groups, surveys, focus groups, observational assessments, 
or a combination of these methods. In general, interviewees from the majority of the 
programs reported positive results from these evaluations, although specific results were not 


supplemented and/or substantiated with reports. 


Only the technology programs reported having content specific evaluations. These 
evaluations were undertaken by a variety of public and private organizations including Dell, 
Intel, and universities. The evaluations tracked the overall effectiveness of the technology 
and larger curriculum, often using computer software to monitor student progress in the day 


school as a barometer to adjust afterschool programming. 
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More than half of the literacy and math programs closely affiliated with the school 
districts did not conduct an external evaluation and approximately one-third of all of the 
programs mentioned having an external evaluator. A significant number of the programs 
considered the process of preparation for the Annual Performance Report (APR/ PPICS) and 


the visits from the state coordinators as a form of external evaluation. 
Internal Evaluation 


All but two programs reported conducted formal or informal internal evaluation. 
Evaluation varied from informal conversations between afterschool staff, day-school staff, 
and parents to a formal administration of surveys to students, parents, staff, and tracking of 
test scores, grades, and attendance records. One program reported a relatively rigorous 
formal evaluation process that involved quarterly assessment reports from staff, pre-post 
testing, tracking of grades and student progress in the day school, and student surveys on 
program satisfaction. The program director explained the intent of the evaluation: 

Once a quarter, each [site] has to come together and present to the Board as well as [the 

director] about what happened last quarter. It’s not just what happened from a laundry 

list. We’re trying to create a learning organization. It’s an opportunity for the unit 


directors to share among themselves and to learn from each other. It creates opportunities 
to share ideas. And it bring a little bit of accountability in. 


Informal evaluations often occurred in staff meetings, i.e., a discussion about what was 
and wasn’t working. One instructor explained, “There is an informal evaluation where we’re 
always talking about what’s working and what doesn’t work and about what we could do 
differently.” At another setting, a project director said that questions such as “Do you feel 
[the program is] effective? Do you think your child has learned?” were addressed to parents 
and teachers to help the program assess itself. Other means of informal evaluation was 
talking to students, parents and day-school teacher feedback, collaborating with peers, and 
anecdotal evidence from individual students who had transformed during or after 


participation in the program. 


In general, interviewees most frequently mentioned using the results of internal 
evaluation to serve as a baseline for instruction, monitor student progress, and to document 
program impact. For example, the principal at one of the science afterschool programs 
reported that student achievement data from an internal evaluation were used to revise 
science curricula at the afterschool program in order to align instruction with the standards 


and improve student performance: 
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That was the reason why the [afterschool program] curriculum went under revision, 
because [our students] were not measuring up nationally. When you look at their 
assessments on the [state test] . . . they weren’t measuring up with other states; and 
because of that we had to go back and revisit our curriculum to see where we were falling 
through the cracks. What we found was that we [needed to revise] our objectives—they 
were too narrow. So teachers were limiting instruction to just those methods, when really 
there was a whole standard that needed to be scored. There was a whole area that 
children needed to know by the end of the school year, so that revision had to take place. 


The community-based programs all reported conducting internal evaluations on a 
regular basis, some more intensive and extensive than others. They appeared to have a 
broader focus, and utilized a mixture of parent, staff, students’, and classroom teachers’ 
surveys to seek feedback. These data served to inform programs from multiple perspectives, 
provided a continuous feedback loop for self-improvement, and measured program 


satisfaction rates. 


A pattern emerged in the resources and purposes of evaluations between the 
community-based programs and the district-affiliated programs. It appeared that the district- 
affiliated programs seemed to have easier access to student academic databases and about 
half of these programs made use of this access to establish baselines and determine areas of 
academic need for their students. The student academic database and classroom grades were 
also used as outcome measures in determining academic growth and progress, especially for 


programs with the explicit goal of raising students’ achievement scores. 
Perceptions of Program Impact 


All programs reported having positive impacts on their students in some ways. Most of 
the evidence provided was anecdotal, but a few formal evaluations and systematical tracking 
of student progress showed improvements in attendance, classroom grades, and achievement 
scores. At the less quantifiable level, several programs reported improvements in attitudes 
toward schooling, student efficacy, confidence, and engagement. Moreover, individual 
success stories highlighted the transformational potential of the afterschool setting in 
improving study skills, attitudes, and behavior, as well as increasing the self-esteem of 


students and their social competencies. 


Most staff interviewed reported that their program had impacted students positively in 
some way. While most feedback on this topic was anecdotal, interviewees most consistently 
cited improvements in the areas of attendance (both in the day school and the afterschool 
program), engagement and interest in the content area, social behavior, and student 


achievement. 
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In interviews, staff indicated that their students’ attendance, both to the afterschool 
program and to the day school, was consistent and showed growth over time. However, few 
of the interviewees provided evidence that linked attendance to achievement. Most of the 
programs also noted that there was a waiting list for their programs, and indicated that there 
were no recruitment or retention concerns. While most respondents mentioned that students 
loved to come to the afterschool program and even preferred attending the afterschool 
program to the day-school program, very few provided enough detail to draw substantial 
conclusions. Nonetheless, afterschool program staff perceptions on program impact are worth 
noting here, and a good number of staff members commented that student interest in their 
afterschool program had facilitated improvements in attendance. One arts instructor noted, 

Attendance is unbelievable. I have kids that say, ‘I only came to school today because I 


knew I was going to be working with you.’ I feel, just from talking with my teachers, that 
behavior problems in some instances are resolved. Students have success in my class. 


Further, the following impressions offered by a project director for one of the science 
afterschool programs indicated a breadth of afterschool program impact not always captured 
by quantitative data: 


I want [the program] to be measured by supporting school success, because that is what I 
think leads to academic achievement. We have clear research that says that the children 
in our afterschool program have better attendance. Their truancy has decreased 
significantly. They’re not tardy. Their behavior in class increases significantly. Their 
connection to a professional adult as measured by the child increases significantly— 
statistically significantly. . . . Those are the things to me that eventually translate into 
school success. So we do think we’re improving academic achievement. .. . We think 
that is significant. 


A technology program director reinforced this assertion and explained: 


There’s a whole part of afterschool that is so hard to put into an evaluation form. That’s 
how you change lives and what you offer, the possibilities and dreams that can come 
true. That’s so hard to put into a question. I think it’s very important that everybody 
understand the power that can come from an afterschool program in affecting change, not 
only cultural change on campus but also individually in particular lives. I have seen a 
whole campus turn around demonstrating that learning can be fun and can still address 
the needs for standardized tests. Their sense of accomplishment that comes from being in 
one of our programs where they have more freedom to explore and have more hands-on 
experiences is profound. 


Some program staff also mentioned improvements in students’ achievement and 


attitudes towards learning. Most of the feedback was clearly anecdotal and related directly to 
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staff perceptions. Yet, some programs were able to cite specific academic gains. For 
example, some of the district-affiliated math and literacy programs observed growth in 
district test scores and/or classroom grades, and many homework staff stated that the 
students’ achievement was greatly related to the homework assistance given at afterschool 
programs. This was corroborated by a day-school principal, 
Our test scores have risen every year for four years. They had gone down every year for 
seven years before [the homework afterschool program started at the school]. In the last 


four years, and including this year again, they’re up anywhere from three to five percent. 
A lot of it is because we started working together. 


All program staff across the content areas also mentioned a marked improvement in 
students’ engagement and interest in the content area over time. Almost all interviewees 
reported that students were happy and enthusiastic to be at the afterschool programs. They 
were eager to attend the programs, and some staff reported that even when the students were 
absent during the school day, they still wanted to come and attend the program. Additionally, 
staff also frequently remarked that students expressed a new appreciation for, and interest in 
the focused content area. For example, one science staff member explained, “I’ve had my 
seventh and eighth graders tell me that they want to go into a career in science instead of 
being a dancer or a football player or something like that. They’ve actually told me they want 
to get a career in science.” Similarly, a staff member of a technology program explained the 
value and relevance of her technology instruction, 

We see a big improvement at the high school. What we’re finding is if you give the 

student the laptop, which we provide for every student to take home and use, they 

complete their homework and it’s higher quality. . .. Even our students that have social 


problems, when they’re more fixed on a project that [uses] the computer, they do a lot 
better. We see a lot of better behavior in those children. 


Consistently, all program staff reported having positive relationships with their 
students, and suggested that the afterschool programs have had positive impacts on students’ 
self-confidence, social skills, and attitudes towards learning. Observation data supported the 
interview data, with students observed to be actively engaged in activities at most of the sites. 
Programs appeared to make specific efforts to select materials that were of high interest to 
students, and program staff engaged students with meaningful questions and frequent 
attempts to get students involved. A majority of the interviewees indicated that their students 
enjoyed and benefited from the afterschool program, and some felt that the afterschool 
programs appealed especially to students who were academically disengaged by offering 


them the kind of personal attention and assistance they needed. 
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Parent Satisfaction 


Parents were surveyed and their input on program satisfaction and perceived academic 
impacts were analyzed. In general, parents reported seeing improvements in their children’s 
skills and overall interest in the specific content area, in their work during the school day, and 
in finishing the specific content area homework on time (see Figure 12). About three-fourths 
of the parents perceived that their children’s skills and interest improved since participating 
in the afterschool program, while a minimal number of parents (1 percent) saw a decrease. 
Approximately 11 percent to 13 percent of the parents did not perceive any changes in their 
children’s performance or attitudes, while an average of 7 percent were not sure about any 
changes. The survey also asked parents about changes in their children’s interest in 


schoolwork in general, and results found that 44 percent of the parents saw a great 


improvement. 
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Figure 12. Parent perception of changes in student behavior. 


Day-School Impact 


Day-school teachers (only in Years 2 and 3 for the arts, science, technology, and 
homework help programs) were asked to describe perceived changes in their students since 
those students began participation in the afterschool program, including any behavioral 
changes. They were also asked to provide their assessments of the changes in students’ 
performance and attitudes. Most teachers, for the 35 programs in Year 2 and 3, reported that 


they noticed positive changes in students’ behaviors in terms of school attendance, frequency 
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of classroom participations, effort on school work, paying attention in class, and discipline 


problems, as displayed in Figure 13 (further detail can be found in Appendix E). 
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Figure 13. Teacher perception of changes in student behavior. 


About 10 percent of the day-school teachers had not noticed a positive change in 
students’ behavior in terms of school attendance, frequency of classroom participation, effort 
on school work, paying attention in class, and discipline problems. About half agreed that 
their students in the afterschool programs had improved in school attendance, were paying 
more attention in class, and had fewer discipline problems. In addition, 58 percent of the 
teachers agreed that the afterschool program students increased their frequency of class 
participation, and most teachers (61 percent) agreed that their students in the afterschool 


program put more effort into school work. 


Day-school teachers were also surveyed in regards to their perception of any academic 
performance and attitude changes in students participating in the arts, science, technology, 
and homework help afterschool programs. Academic performance and attitude changes were 
defined differently according to the specific content area. The definitions for the four content 


areas were as follows: 


66 


¢ Arts: willingness to participate in various arts activities, the overall quality of their 
art projects, perceptual and expressive skills, technical skills (e.g., ability to paint, 
sing, play a musical instrument), attitudes toward the arts in general, and ability to 
engage in higher order thinking. 


¢ Science: science skills, science test grades, attitudes toward science and homework, 
and the quality and timely completion of science homework. 


¢ Technology: participation in technology activities, overall technical skills, attitude 
towards technology, engage in higher order thinking, cooperative collaborative 
skills, persistence in school work, and problem-solving skills. 


¢ Homework: motivation for completing homework, ability to complete homework, 
time management skills, organizational skills, cooperative and collaborative skills, 
performance on tests, persistence in school work, problem-solving skills, study and 
research skills, and quality of work overall. 

Most notably in the four content areas, a majority of teachers perceived an 
improvement in their students’ overall interest and skills in the content-specific area. Most 
teachers also noticed a positive increase in their attitudes towards the content area. Perhaps 
due to the fact that some of the teachers surveyed did not directly supervise the students in 
arts, science, or technology, up to 30 percent of the teachers responded that they did not 
know whether there had been a change in the students’ academic performance. Meanwhile, 
of the teachers whose students participated in homework assistance afterschool programs, 47 
percent to 75 percent perceived improvements in their students’ performance on tests and 


their ability to complete homework. 
Summary of Evaluative Structures and Program Impact 


In summary, external evaluations on the content-specific component of the programs 
were not always conducted. Only technology programs consistently reported having external 
evaluations on the technology component of their programs. In other programs, if external 
programs were conducted, they were usually program wide where the findings were 
generally positive and strong evidence to support the claims was rare. Most of the programs 
conducted internal evaluations, some more formal than others. The purpose of these 
evaluations was for program improvement, such as monitoring student progress, measuring 
program growth, research for program needs, and gauging program impacts. Except for some 
of the district-affiliated math and literacy programs, student assessments throughout the 
programs were informal, generally in the forms of spot checking, questioning, and observing 


student understandings. 


Overall, parents, staff, and teachers perceived programs to have positive impacts on 


student engagement, attitudes, skills, and their knowledge. More specifically, programs were 
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perceived to improve attendance, especially if the students were engaged in the afterschool 
programs. Students appeared happy and enthusiastic to be attending the program, and 
engaged in program activities. All afterschool staff reported having positive relationships 
with their students, and consistently suggested that the programs had positive impacts on 
students’ self-confidence, social skills, and attitudes towards learning. Most of the program 
staff mentioned an improvement in students’ achievement and attitudes towards learning. 
Day-school teachers reported that they perceived improvements in the students’ attitudes and 
skills toward four of the six content areas. Behaviors (i.e., school attendance, frequency of 
class participation, effort on school work, paying attention in class, and fewer discipline 
issues) were also perceived to have greatly improved. In all of the content areas (except 
homework), parents also perceived improvements in content-specific skills, interest in a 
specific content area, interest in schoolwork in general, and finishing content-specific 
homework on time. Parents with children in the homework help programs saw improvement 
in their child’s motivation to complete homework, ability to complete homework, time 


management skills, organizational skills, and interest in school work in general. 
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CHAPTER 8: 
IMPLICATIONS AND CONCLUSION 


The findings in this study indicate that the characteristics of the identified afterschool 
programs appeared to be aligned with the criteria established for high quality practices in 
terms of program structure, student engagement, instructional strategies, and issues of 


accountability which served as the basis for the development of the logic model. 


The common characteristics that were found across the afterschool programs are listed 
here: 

» These programs had strong leadership and established clear goals 

» They aligned program structures and contents to meet goals 

» They set a schedule that allowed time for students to learn and practice 

» They established relationships with the day school 

» Their curriculum in general reflected a linkage to standards 

«» Most of the programs used research-based strategies 

» They all maintained some form of evaluative structures 

» Staff members had low turn over rates 

» Staff members related well to the students 

» The staff were able to build rapport, maintained high expectations, kept students 

motivated and engaged. 

Examples of these common characteristics follow. To increase the students’ academic 
skills, many of the math, literacy, and homework programs used a “seamless approach” to 
connect day school with afterschool and employed certified teachers. To expose students to 
the experiences of science, arts, and technology, programs used hands-on approaches and 
generally collaborated with organizations strong in the specific content areas, such as Arts 
Corps, Girls’ Inc., local universities, Dell, Apple, etc. Almost all of the 53 programs 
maintained good relationships with the day-school personnel. However, few of them had 
established formal communication systems for that purpose. Across all the programs, staff 
had knowledge of content standards and engaged in content specific research-based 
instructional strategies. To engage students and staff, these programs recruited qualified staff 
and created working environments that built relationships and inspired support for the 
programs’ missions. Because staff felt respected, supported, autonomous, and confident in 
their ability to reach their students, these programs were able to retain staff and achieve lower 


turnover rates than other programs. In turn, staff and students were able to construct positive 
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relationships with each other, characterized by warmth and mutual respect. This supportive 
relationship permitted the staff to serve as role models, creating a norm of high expectations 
and strengthening student behavior, school attendance, work habits, and attitudes towards 
learning. At the same time, most programs also recognized the importance of considering 
student voices when making decisions regarding program activities and content; hence, many 
programs allowed students to provide input, especially in the arts and technology programs. 
Many of these programs also made use of research-based strategies to engage and motivate 
students, making learning fun. Overall, programs reported some form of internal and/or 
external evaluations for accountability and self-improvement purposes. As a result, students 
seemed engaged and excited to be in the programs. In general, day-school teachers, parents, 
and afterschool staff perceived the programs to have positive influences on the students. 
Some programs reported increased school attendance, better test scores, and most of the 
programs reported positive changes in behaviors and attitudes towards learning. Parents were 
very satisfied with the programs both in terms of positive changes in their children’s 


behaviors and attitudes, and in general program functioning. 


As discussed earlier, the single most dominant characteristic that was consistently 
revealed in all programs was the high motivation level of the sites. These high functioning 
programs were characterized by strong leadership. Staff members were empowered to engage 
in program missions that included internal motivations, such as making a difference in the 
lives of their students. It appeared that this intrinsic motivation for staff members to bond 
with the students was a powerful force toward the aspirations of both the staff and the 
students. This relationship can be further enhanced, so that the afterschool staff can foster the 
relationship to induce academic resiliency in their students. As a result of the findings, there 


are numerous implications for high quality afterschool practice. 
Implications for High Quality Afterschool Practice 


As the quality of afterschool programs is further embedded in the social systems of 
schools, families, communities, networks, state, and federal policies and regulations, all will 
have to function collaboratively together to induce sustainable change in students. Since key 
findings of this study point to the strength of strong leadership, alignment to the day school, 
and the motivational value of the staff-student relationships, there is the potential for 
enhancing staff skills to create a tighter social system and bring in more resources from the 
day school, families, and communities. In order to do that, there is the need to provide 
meaningful, informative, and regular professional development opportunities to the 
afterschool staff to further enhance their abilities and collaborative skills. However, as the 


findings of this study indicate, the participation rate in professional development is quite low. 
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Therefore, additional incentives through policies and guidelines are needed to encourage staff 
to take part in the professional development. These can be achieved through: (a) policies that 
support staff (professional) development, (b) funding sources and designations, and (c) 


ongoing program evaluation to prescribe professional development needs. 
Professional Development 


The study’s findings show that professional development was not regularly offered at 
all programs, and when offered, participation rates tended to be low. Moreover, project 
directors and site coordinators appeared to attend conferences and workshops more often 
than the program staff. This may be due to a number of factors, including the higher turnover 
rate of other staff members (in comparison to project directors and site coordinators), the 
tight scheduling in afterschool programs (which makes it difficult to spare staff members for 
professional development), and limited budgets to pay staff time for professional 
development or to pay for substitutes. Since professional development has a strong 
correlation with staff efficacy, quality of the delivery of the intended lessons, and hence 
student outcomes (Ingvarson, Meiers, & Beavis, 2005), federal and state policies guiding the 
development and operations of afterschool programs should take the study findings into 
consideration and provide additional guidelines and regulations for the provision of sufficient 


quality professional development opportunities for afterschool programs. 


At the site level, program directors and site coordinators should provide professional 
development opportunities to increase staff readiness to address student needs. Program 
directors should re-examine program goals to decide whether site coordinators or program 
staff would benefit most from attending certain conferences and professional development 
opportunities, and make decisions accordingly. If there is a “training-of-the-trainer” approach 
at the sites, a systematic approach or manual should be established to ensure professional 
development fidelity. When planning the yearly calendar, time off for professional 
development, reserving substitutes for the staff to attend professional development, or 


closing the program a few times a year to allow for staff development should be considered. 


The findings also point to several specific areas where afterschool staff would benefit 
from professional development: (a) content and curriculum, (b) collaboration with the day 
school, (c) evaluation and assessment, and (d) parent involvement. While most program staff 
were aware of the standards within specific content areas, they were less knowledgeable 
about the links between the standards and their instructional practices. To support program 


staff in becoming more familiar with curriculum standards and to be more creative with the 
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implementation of these standards in their instruction, professional development in content 


and curriculum should be consistently provided. 


While discussing relationship building and curriculum content and standards, it should 
be noted that the positive relationships that existed among the students and staff may have 
been due to a variety of unexplored variables that researchers are not yet able to affirmatively 
identify (e.g., that the afterschool staff were from the same community; the afterschool staff 
shared similar life experiences as the students and were able to connect to them better; a 
certain aspect of their personality made them more appealing and trustworthy to the students, 
etc.). Thus, one should not be hasty and take the short route in replacing current staff with 
certified staff in order to promote better academic outcomes, instead professional 
development and enhancing the readiness and knowledge base of the current staff would be 


the recommended route. 


To facilitate this practice, state agencies should set policies to guide program 
functioning and provide periodic monitoring to ensure that staff members receive adequate 
professional development to meet program expectations for high quality, engaging academic 
instruction. State agencies should also work closely with professional development providers 
and afterschool programs to coordinate the professional development topics offered at 
national and regional conferences to meet staff needs and interests. With the efficiency of 
technology, toolkits such as the ones offered by the National Partnership, make it possible to 
instantly deliver to any program the most up-dated instructional practices in a cost-effective 


way, and these should be utilized at all program levels. 


Furthermore, when making policies regarding afterschool programs, policy makers 
should keep in mind that even with the best pre-approved, flawless research-based 
curriculums, the delivery of the content is as important as the content itself. Thus, policies 
and legislation guiding program certifications and recertifications should consider the need 
for staff members and administrators of afterschool programs to maintain continuous 


professional development, updates and support with the new developments in the field. 


The role of the day school in professional development is also critical The study 
findings indicated that the day school plays an influential role in the efficiency of their 
affiliated afterschool programs. The day school can provide space (e.g., classrooms, 
playgrounds), and resources (e.g., use of computers, academic and behavioral data on the 
students, access into classroom records, professional development opportunities, etc.) that are 
important to afterschool program organization and student learning. All of the programs 


visited in this study maintained good relationships with the day school. However, as 
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important as this relationship is, very few programs in this study had set up strategic systems 
to establish and strengthen this relationship. Professional development can assist by 
identifying procedures to establish channels of communication, and identifying systematic 
approaches for after school programs to make contacts and collaborations (e.g., 
communication logs, planners, email, meetings, and ideas for developing more collaborative 
program structures like overlap in day school and afterschool program staff, offering 
afterschool program staff paid time to communicate and work with day-school staff, and 
administrative support in creating connections to the day school). 


Furthermore, state and local educational agencies can organize professional 
development opportunities for local afterschool programs to share their experiences with 
each other. State and federal program officers may even want to participate in specific 
professional development for the delivery of specific curricula together with their programs, 
so that they can have a realistic expectation of what the program can achieve and help set 


appropriate benchmarks. 


Evaluation and assessment is another area to be considered for changes to afterschool 
staff development. As outlined in the logic model, continuous evaluation is necessary to 
gauge program performance and strive for continuous improvement. However, across the 
programs, evaluation was mostly used to gauge program impact and monitor student 
progress. Using evaluation as a reflection to guide program improvement was not practiced 
as frequently, especially at the staff level. Professional development could stress the 
importance of this continuous feedback loop for the sustainability of program quality, and 
provide programs with the skills and tools to jump start this learning cycle. First, afterschool 
staff would need to develop strategies on how to locate/develop reliable self-assessment tools 
that can measure program growth and improvement. Once programs have the tools to assess 
program results, professional development should be provided on how to interpret findings 
and translate assessment results to improve afterschool programming and instructional 
practices. In addition, student assessment and tracking of student progress should also be a 
priority in professional development. Through regular monitoring of student work and 
achievement, afterschool programs can have a better understanding of the needs of their 


students, and areas of support that need to be improved. 


Parent involvement is another area that can be enhanced in afterschool professional 
development. While the research literature continuously stresses the importance of parent 
involvement in influencing children’s academic outcomes, this study repeatedly found that 
parents, though very satisfied with the programs, were generally not involved in attending 


events or volunteering in afterschool programs. This finding is consistent with other literature 
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on afterschool programs (Huang et al., 2007). These results suggest that perhaps perceived 
parent involvement and parent expectations are just as important as actual attendance at 
events. While some parents may be less able to attend afterschool events due to 
circumstances such as long work hours, other children in the household, being a single 
parent, and not knowing how to be involved, parents may be highly involved in other ways, 
such as giving their children access to educational resources (e.g., books, computers) and 
emphasizing the importance of a good education. Some of the culturally specific ways of 
being involved may be just as important as attending school events especially for immigrant 
parents and parents who feel less efficacious in dealing with the educational system (Zhou & 
Kim, 2006). 


Despite the structural constraints families conceivably face, professional development 
in the area of parent involvement can focus on training staff to view parent involvement as a 
broad set of behaviors that could involve not only attending school events, communicating 
with the afterschool staff, but also in providing resources to their children for learning, 
holding high expectations and aspirations for their children’s education, developing and 
employing social networks to gain information and links to educational opportunities, and 
providing assistance in their child’s school work at home. In understanding and supporting 
different styles of parental involvement, staff can build partnerships in various forms with 
parents in supporting the students. Furthermore, professional development can also be 
helpful in providing staff with ideas on how to develop rapport with parents, such as focusing 
on student strengths, maintaining frequent communication and contact, offering suggestions 
and strategies for different forms of parental involvement, emphasizing the importance of 


parent involvement, and giving parents a voice in afterschool programming decisions. 
Afterschool Funding 


Another area that can have important influences on the quality of afterschool 
programming is funding sources. In writing funding contracts, funding agencies can include 
categories in the budgets specified for professional development, staff stability, and 
involvement with the day school and parents. Programs can ensure practices that include 


developing sustainability plans and collaborations to secure future funding when grants end. 


As mentioned in the previous section, professional development is a necessity for 
afterschool staff. Funding agencies should create specific budgets for continuous staff 
development as a way to ensure that content staff will be adequately prepared and 
continuously updated with new information. If the goal of funders is to target professional 


development for specific staff members such as instructors, site coordinators, or coaches, it 
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should be clearly specified in the contracts for afterschool programs. When allocating 
funding for training and professional development, the costs of staff time and the cost for the 
substitutes for the staff members attending professional development should be considered 


together with the overall cost of the training. 


Staff stability is another funding issue that impacts the quality of afterschool programs. 
The study results indicated that at high functioning programs, staff turnover was considered 
to be low. More than 60 percent of the staff had at least 1 year at their site and over 30 
percent had over 4 years of experiences at the site. Staff stability is important for relationship 
building, since staff members serve as constant role models and mentors in their students’ 
lives and provide the basis for students to build trust, positive attitudes and efficacy toward 
learning. To reduce staff turnover in the afterschool field, funders and afterschool 
administrators should consider incentives for building morale such as rewards (e.g., further 
educational opportunities such as a grant for tuition). One suggestion would be reward an 
afterschool team/teacher nominated by parents, teachers, or the students. Recognition of the 
outstanding performance of specific staff, a site, or program may also offer 
acknowledgement for high quality performance. A pay scale incentive for years of service 
and a possible career ladder would also help staff retention. Policy makers and legislators 
may consider retention plans that establish policies and legislation to recognize program 


workers’ contributions to student success. 


Funding issues related to the involvement of the day school and parents with the 
afterschool program is also related to the quality of the program. In order to ensure linkages 
between the afterschool programs and day school, some formal agreements should be made 
prior to the inception of the grant/contract writing. For example, the administration at the day 
school can connect to certain conditions and resources, such as granting the use of certain 
facilities at the school, providing time for school teachers and afterschool staff to meet and 
plan lessons together, establishing a system of communication, etc. (e.g., homework log 
between day teachers and afterschool staff). Funding agencies can budget additional 
resources for afterschool programs that would facilitate these linkages with school. For 
instance, the budget may include a person designated as the day-school liaison who can 
establish a chain of intentional communication and collaboration with the day school. 
Additional funding for relationship building such as staff retreats or workshops (together 


with day-school teachers) can also be a way to form collaborative relationships. 


Similarly, to develop relationships and opportunities for parent involvement, funding 
agencies should budget for parent participation, time for staff to communicate and 


collaborate with parents, and resources for families. Funding allocated to providing childcare 
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during parent events, (e.g., ESL, parenting, other educational classes, Open Houses, parent- 
teacher conferences) will facilitate parent participation. Home visits and family assistance 


can further solidify the relationships between the afterschool and the families. 


The importance of parent involvement specific to afterschool programs and student 
outcomes should be studied further and more rigorously. If the importance of the 
relationships is confirmed, policies can be set to encourage parents to make certain 
contributions of their time when their children enroll in the program. It should be noted that 
parents’ contribution of time should not always require time spent at the program site; a 
project that the parents work on with their children at home can also be considered as parent 


involvement. 
Continuous Evaluation and Improvement 


Continuous evaluation and improvement are important components of high quality 
afterschool programs. Implications of our findings in this area suggest a need for more 
systematic evaluation, enforced through local and federal policies and regulations. 
Specifically, the use of evaluation tools in program planning, curriculum development, and 
general improvements can lead to improved student outcomes. Standards-based curriculum 
and research-based instructional strategies are key components of high quality afterschool 
programs. To ensure that students are benefiting from the curriculum and instruction, 


periodic student assessments and program evaluations ought to be conducted. 


The findings indicated that student assessment was not a common practice across the 
high performing programs studied. In general, at the 53 programs, student assessments were 
characterized as informal, including spot checking, questioning, and observing students’ 
understanding. Qualitative findings further revealed that many staff and site coordinators 
were not familiar with assessment and evaluative tools, and did not have a clear 
understanding of their purposes. At the afterschool classroom level, periodic assessment 
should be conducted and focused on monitoring student progress and achievement. 
Professional development can provide opportunities for afterschool staff to review and 
understand the purposes and importance of assessment and evaluations in relation to student 
outcomes. Reliable and simple evaluation and assessment tools need to be developed and 
used regularly by the afterschool classroom and administrative staff. Program staff can also 
be trained and encouraged to conduct self-assessments to understand their own knowledge of 
the curriculum, and their strengths or weaknesses in implementing the curriculum through 
research-based practices. By implementing more formal and periodic student evaluations, 


afterschool programs can better gauge student learning and strategically connect student test 
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scores to the curriculum, as well as make programmatic improvements to better meet student 


needs. 


At the program level, program directors and site coordinators need to clearly define the 
purposes of evaluations, so that program staff can actively participate in them. For example, 
assessment information can be used for self-improvement in program planning and 
curriculum development, other internal evaluations can be used to guide relationship 
building, administrative procedures, and resource allocation. Through periodic and 
systematic evaluations, afterschool staff can be made aware of their areas of strengths and 
weaknesses, and make informed decisions. For instance, under the topic of staffing, self- 
evaluation tools can be used to understand staff professional development needs, staff 
utilization of research-based activities, and staff knowledge of standards-based curriculum. 
Using these assessment results, program directors can implement changes, allocate resources, 
and design professional development opportunities to further staff expertise in the needed 


areas. 


In addition, multiple perspectives should be sought when gathering evaluation data. 
Programs need to be encouraged to request parent, student, and community input when 
evaluating program components. The results of these evaluations ought to serve more than 
accountability needs, with the interpretation of the findings also guiding continuous program 
improvement. Consequently, not only will evaluation help programs run smoothly, but it can 
also help build positive relationships with parents, students, and community members by 


giving them voice in program planning and improvement. 


Furthermore, state and federal guidelines can be established to support programs in 
their evaluation process. Policymakers can require programs to engage in periodic 
evaluations, document their progress, identify their strengths and weaknesses, and submit a 
plan for change (such as curriculum modifications and adaptations to meet student needs and 
enhance the program outcomes) in addition to the programs’ accountability/monitoring 
reports. As part of the guidelines, a system for interpretation of program results ought to be in 
place for programs to structure their findings and reset their goals each year. Administrative 
staff should be given opportunities to be trained in conducting self-assessments of their 


programs, as well how to translate assessment results into program improvement plans. 


State coordinators can also benefit by participating in professional development around 
evaluation. This development may provide additional efficacy and understanding about the 
functioning of afterschool programs and create further clarity when the coordinators review 


the programs’ periodic reports. The state coordinators can make use of the findings in the 
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reports to monitor and initiate conversations with the programs, building rapport with the 
programs by providing support and resources during the programs’ development as part of 


the collaborative efforts. 
Contributions of the National Partnership 


One of the major goals for this 5-year study was to offer strategies, tools, and technical 
assistance to help address two continuing challenges to afterschool programs: (a) ensuring 
that programs offer high quality, research-based academic content utilizing appropriate 
methods of teaching and learning; and (b) ensuring that programs are able to attract and 


retain students who participate regularly and thus can benefit from these investments. 


Other than the contributions that this report brings to assist afterschool programs to 
overcome these challenges, the National Partnership synthesized the study findings on 
effective practices to develop on-line Afterschool Toolkits in the six content areas (math, 
literacy, science, arts, technology, and homework help). These Toolkits provide assistance to 
programs in ensuring high quality, standards-based academic content, and in utilizing 
research-based teaching and learning strategies. Through the use of the Afterschool Toolkits, 
concrete strategies are provided to help practitioners build program and staff capacity. The 
Toolkits offer suggestions, illustrations, and demonstrations on how programs can embed 
academic learning in all aspects of afterschool activities. During the 5 years of this contract, 
the Toolkits have been consistently revised and updated as data and findings from the 
National Partnership became available and as the broader research literature evolved. In 
addition, professional development on the use of the Toolkits has been continuously provided 
in national and regional afterschool conferences and workshops. Feedback gathered from 
these professional development events has also been used to continuously improve the 
Toolkits. The National Partnership maintains a web site that houses the Toolkits. They are 


available at the site: www.sedl.org/afterschool. 


In addition to the study report and the online Afterschool Toolkits, a separate 
assessment tool guide,'* based on the instruments constructed for this study, was developed 
to assist afterschool programs in self-assessment, both at the program organization level, and 
the content (math, literacy science, arts, technology, and homework) instructional features 
level. With current policies and legislation that point to the accountability of Extra Learning 
Opportunities (ELO), the National Partnership’s self-assessment guide provides practitioners 
a vital resource for self-evaluation. This toolkit includes instruments for programs to examine 


their curriculum content and instructional practices in the six content areas, and offers 


'? The instruments for the self-assessment guide are a separate report to the U.S. Department of Education. 
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general guidelines on effective program management and administration. It will help 


programs to make informed decisions on developing strategies to attract and retain students. 


The overall contribution of the National Partnership is illustrated in the Figure 14. 
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Figure 14. The Toolkits and the Continuous Self Improvement Model. 
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As illustrated in Figure 14, with the Toolkit, the National Partnership offers self- 
improvement opportunities for afterschool programs in all essential program components: 1) 
curriculum design, 2) instructional strategies, 3) opportunities to practice (e.g. suggestions 
for project durations, follow-up activities, project extension possibilities), 4) periodic 
assessments, and 5) use of the results of the assessments to fine-tune the program goals. In 
the future this system can be further expanded and enhanced to include professional 
development and instruments on goal setting and provide a set of validated rubrics (perhaps 
also an analytical software to provide simple statistical results for the programs) so that 
education agencies, funding agencies, and policy makers can use the toolkits to establish 
standardized expectations for afterschool programs. At least once a year, afterschool 
programs should take stock of their progress and make plans for refinement. This process 


will empower sites to make grounded decisions and to improve their program performance. 
Study Limitations 


While the findings of this study have implications for afterschool programming and 
future research, a number of limitations should be noted. First, the program participants in 
this study were carefully selected and represented only a small portion of afterschool 
programs in the nation. Thus, the findings on this sample cannot be generalized to the 
afterschool field as a whole. Secondly, since this study sought to conduct extensive site visits 
and intensive data collection on programs that focused on multiple content areas, there were 
time lags between Year 1 and Year 3 of data collection. Coincidently, this was a period of 
tremendous development and growth, while afterschool activities were generating nationwide 
attention in education and political arenas. This drastic growth could have contributed to the 
differences in program content, structure, and practices across programs over the years. 
Audiences should keep this in mind while interpreting results from one year to the next. 
Finally, the purpose of this study was to identify the indicators of high quality afterschool 
programs within the six content areas. Due to the differences in context, a discontinuity of 


the study design was observed which makes any longitudinal comparisons more difficult. 
Conclusion 


The National Partnership has successfully accomplished the five major tasks for this 
project aimed at improving the delivery and quality of academic content, teaching, and 
professional development in afterschool programs. Fifty-three high functioning programs 
representative across eight regional divisions of the nation, including rural and urban 
programs, community-based and school district related programs, were identified using 


rigorous methods. High quality practices in program organization, program structure, and 
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especially in content delivery were studied. The findings were synthesized into the 
Afterschool Toolkits that was made available to programs nationwide via the world-wide- 


web. Professional development was conducted consistently and extensively throughout the 
nation. 


Stories, such as the one below for students in one of the technology programs visited in 
this study, clearly indicates that quality in afterschool programs can potentially lead to 
success for students and society: 

XXX kids have become leaders in the day school because of their advanced technology 

skills and increased confidence [gotten from the afterschool program]. XXX kids 


represented the school at the State Superintendent’s Technology Day at the State Capitol 
this last winter!” (described by a site coordinator in an interview) 


Technology is a part of the environment at this program. The program’s goal is not to 
“teach technology” but to use technology to engage students in their learning and to help the 
students to become more successful in school. They have certainly aligned and structured 
their resources and achieved their goal! 


Using the Toolkit and self-improvement model (as shown earlier in Figure14) to self- 
monitor and enhance the program environment, curriculum content, and instructional 


strategies, all programs will have the capacity and potential to achieve their goal. 
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Appendix B 


Afterschool Program Staff Work Experience 


Table Bl 
Staff Experience at Math Programs 


Years of ASP experience Site coordinators Instructors 
N=6 N=37 
Average years at current site 4.8 3.4 
Average years at ASPs in general 5.0 3.5 
Table B2 


Staff Experience at Literacy Programs 


Years of ASP experience Site coordinators Instructors 
N= 15 N= 110 
Average years at current site 4.1 3.6 
Average years at ASPs in general 4.6 3.8 


As improvement of the study instruments, the scale for reporting work experiences for 
arts, science, technology, and homework were restructured. Percentages were calculated for 
each time periods for better representations. 


Table B3 
Work Experiences of Art Staff 


Length of time working In general % At current site % 
Less than a month 00 03 
a few months 10 31 
a year 07 00 
1-3 years 33 44 
4-7 years 20 19 
more than 7 years 30 03 
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Table B4 


Science Work Experience 


Length of time working Instructors Instructors 
Less than a month 0% 0% 
A few months 12% 17% 
A year 6% 6% 
1-3 years 38% 30% 
4-7 years 36% 26% 
More than 7 years 8% 22% 


Table BS 
Work Experiences of Technology Staff N = 16 


Length of time working In general At current site 
A few months 13% 13% 
A year 6% 13% 
1-3 years 19% 38% 
4-7 years 50% 31% 
More than 7 years 13% 6% 
Table B6 


Work Experiences of Homework Instructors N = 50 


Length of time working In general At current site 
A few months 14% 20% 
A year 8% 18% 
1-3 years 36% 32% 
4-7 years 28% 30% 
More than 7 years 14% 0% 
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Appendix C 


Standards-based Instruction 


Table Cl 


Percentage of Instructors’ Use of Standards-based Literacy Instruction 


Literacy Contents Yes No 

K-2 92% 8% 
Print representation of spoken language. 
Recognizing common text features such as headings, key words, charts. 89% 11% 
Demonstrating awareness of sound-symbol relationships. 88% 12% 
Understanding alphabetic principles (that each letter represents a sound). 89% 11% 
Applying decoding to comprehend text (e.g., breaking apart words to understand 92% 8% 
meaning). 
Reading silently or aloud with fluency (smoothly and easily). 96% 4% 
Self-monitoring/self-correcting reading. 85% 15% 
Vocabulary development. 96% 4% 
Identifying literary devices (e.g., simile, metaphor). 82% 18% 
Understanding antonyms/synonyms. 82% 18% 
Grades 3-5 
Using glossaries, table of contents, chapter headings, and indexes to locate information. 82% 18% 
Applying phonetic strategies to make meaning from text. 91% 9% 
Applying decoding to comprehend text. 91% 9% 
Developing pre-reading strategies. 91% 9% 
Understanding textual features. 82% 18% 
Understanding prefixes, suffixes, and affixes. 85% 15% 
Using pictures and context cues to understand meanings of words. 91% 9% 
Identifying homophones and homographs. 82% 18% 
Understanding story components. 88% 12% 
Self-monitoring for comprehension. 94% 6% 
Making inferences using evidence. 88% 12% 
Reading a variety of literary genres. 89% 11% 
Researching topics using a variety of materials. 74% 27% 
Grades 6-9 
Using text features such as lists, indices, headings. 69% 31% 
Identifying/using text organizational structures to gain meaning from text. 72% 28% 
Applying self-correcting strategies to decode text. 55% 45% 
Making predictions/drawing conclusions. 79% 21% 
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Literacy Contents Yes No 

Self-monitoring for reading. 76% 24% 
Vocabulary development. 83% 17% 
Identifying figurative and literary devices. 75% 25% 
Analyzing the purpose of different literary texts. 61% 39% 
Understanding literary techniques. 61% 39% 
Developing and investigating research questions. 61% 39% 

45% 55% 


Producing book reports or other written projects. 
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Table C2 


Percentage of Instructors’ Use of Standards-based Math Instruction 


Math Contents Yes No 

K-2 92% 8% 
Concepts of numbers (e.g., whole numbers, ordinal and cardinal numbers, fractions). 
Addition and subtraction of whole numbers. 100% 0% 
Pattern recognition (e.g., through sorting/classification of objects or sounds). 100% 0% 
Basic conventional math symbols (e.g., plus and minus sign, equals sign). 100% 0% 
Basic spatial relationships (e.g., drawing and describing objects). 100% 0% 
Basic concepts of change (e.g., understanding that amount of change can be quantified). 100% 0% 
Basic measurement concepts (e.g., length, volume, weight, area, time). 85% 15% 
Grades 3-5 
Understanding/application of fractions, decimals, percentages. 93% 7% 
Multiplication and division. 94% 6% 
Understanding of patterns (using numbers or shapes). 93% 7% 
Using equations to express relationships between numbers. 100% 0% 
Using graphs, tables, or other graphic representations. 100% 0% 
Working with 2- and 3-dimensional shapes. 88% 12% 
Basic measurement (length, area, weight, volume). 88% 12% 
Designing studies and collecting data. 88% 12% 
Grades 6-9 
Using fractions, decimals, and percentages. 50% 50% 
Using ratios and proportions. 50% 50% 
Using tables or graphs to represent/analyze problems. 50% 50% 
Understanding/applying basic geometric concepts such as angles, side length, perimeter, 50% 50% 

area. 
Formulas for areas of more complex shapes (e.g., triangles, parallelograms, trapezoids, 50% 50% 
circles, pyramids, cylinders). 
Problem solving using equations. 50% 50% 
Designing small research studies. 100% 0% 
Understanding/applying mean, range, and median. 50% 50% 
Representing data in charts, such as histograms, scatter plots, or box plots. 100% 0% 
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Table C3 


Percentage of Instructors Use of Standards-based Arts Instruction 


Arts content Yes Don’t know 
Create arts experiences expressing self or environment. 85% 8% 
Apply art techniques and processes. 93% 0% 
Make connections to history and culture. 67% 15% 
Engage students in analyzing and communicating. 82% 4% 
Integrate the arts with other subjects. 58% 27% 
Use technology as a creative tool. 33% 15% 
Listen to, analyze, describe music. 59% 15% 
Sing in groups or individually. 67% 15% 
Play a musical instrument. 41% 15% 
Learn skills in theatre arts performances. 59% 7% 
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Table C4 


Percentage of Instructors’ Use of Standards-based Science Instruction 


Don’t 
Science content Yes know 
Understanding concepts of scientific inquiry 71% 25% 
Properties / position / motion of objects. 60% 38% 
Light, heat, electricity, magnetism. 54% 39% 
Characteristics and life cycles of organisms. 68% 26% 
Position and motion of objects. 57% 38% 
Objects and changes in earth and sky. 56% 33% 
Understanding of science/technology. 80% 18% 
Personal health and nutrition. 77% 16% 
Changes in populations and environments. 58% 36% 
Changes in properties in matter. 66% 30% 
Motions and forces. 65% 29% 
Transfer of energy. 60% 33% 
Structure / function of living systems. 58% 30% 
Reproduction and heredity. 33% 54% 
Regulation and behavior. 48% 44% 
Populations and ecosystems. 49% 35% 
Diversity and adaptations of organisms. 52% 33% 
Structure of earth's system / solar system. 62% 27% 
Populations /resources and environments. 51% 41% 
Natural hazards, risks and benefits. 50% 44% 
History of nature and science. 46% 40% 
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Table C5 


Percentage of Instructors’ Use of Standards-based Technology Instruction 


Technology contents Yes Don’t know 
Applying strategies to hardware and software. 60% 13% 
Understanding the nature and operation of tech. systems. 87% 7% 
Using tech while working independently. 81% 13% 
Using tech to process data and report results. 50% 21% 
Using tech. to collaborate with peers on projects. 63% 19% 
Using telecom to collaborate with peers and others. 79% 0% 
Using tech. resources for real-world problems and concerns. 80% 0% 
Using tech as a tool for creative projects. 87% 7% 
Using tech to locate, evaluate, and collect info. 88% 13% 
Evaluating and selecting tech tools. 73% 20% 
Practicing responsible behavior and use of tech. 100% 0% 
Discuss consequences of misuse of tech systems. 86% 7% 
Research and evaluate the accuracy and bias of electronic info. 43% 36% 
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Appendix D 


Instructional Strategies 
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Figure D1. Frequency of specific instructional practices in math program. 
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Figure D2. Frequency of specific instructional practices in literacy program. 
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Figure D4. Frequency of specific instructional practices in science program. 
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Figure D5. Frequency of specific instructional practices in technology program. 
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Figure D6. Percentage of homework instructors engaged in strategies. 
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Appendix E 


Day-School Teacher Perception of Changes (Figures E1—E4) 
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Figure E1. Teacher perception of changes in student performance and attitude toward arts. 
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Figure E2. Teacher perception of changes in student performance and attitude toward science. 
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Figure E3. Teacher perception of changes in student performance and attitude toward technology. 
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